If you are using the overlay2 storage driver, you can place limits on the rootfs within a container but only if using an xfs backing filesystem (not ext4). As a quick test of your Docker install, check your Docker storage driver and backing filesystem, then attempt to spin up a small alpine image with a … Docker: Use overlay2 with an xfs backing filesystem to limit rootfs size
The physical partitions and filesystem formats on your host are configured for your main workload, but if you want an application to use a specific filesystem (xfs, ext4, zfs) and size capacity without reconfiguration at the physical level then you can consider a loopback image. For example, if we create a 100Mb disk file named … Linux: Mounting a loopback ext4/xfs filesystem to isolate or enforce storage limits
XFS is a journaled filesystem that has excellent parallel performance, and is licensed under the GPL which means it has been included in many Linux distributions. One of the features of XFS is the ability to enforce quotas based on user, group, and project. In this article, I will show how to assign filesize quotas … Linux: Using xfs project quotas to limit capacity within a subdirectory
If you are looking for basic HTTP file uploading using Flask, you can use the code in my python-flask-upload-files project on github as an example. This project exercises multiple scenarios: Upload a single file in chunked mode (not forced to save entirety to disk) Upload a single file Upload multiple files Error message shown when … Python: Using Flask to upload files
If you need to call attention to a message within your script, a splash of color is often the best way to get noticed in the sea of monochrome text. The standard ANSI escape codes are supported at most consoles, and in its simplest form can be used like this: RED=’\033[0;31m’ GREEN=’\033[0;32m’ NC=’\033[0m’ echo -e … Bash: Outputting text in color for readability
This may come as a surprise, but Bash only supports integer arithmetic natively. If you need to perform calculations on floating point numbers, you will need to call out to a utility program like bc or Python. As a quick proof to yourself, try multiplying integers, then do the same with floating point numbers. $ … Bash: Performing floating arithmetic using bc
If you are using Flask to generate dynamic content of significant size, such as large binary images/pdf or large text-based datasets, then you need to consider streaming to minimize the memory footprint of Flask and preserve scalability. Using an inner generate function and a yield allows Flask to return chunks of data back to the … Python: Using Flask to stream chunked dynamic content to end users
Update Jan 2022: If you would rather have a mail server running in Kubernetes, see my article here. There are many reasons you might want your own Postfix server: maybe you need to test various relay settings, or validate the ability of your mail script to use TLS, or maybe you are a developer that … Docker: Running a Postfix container for testing mail during development
Although sending a plain text message through an unauthenticated SMTP relay might be easily done with a few lines of Python code, sending rich HTML email through authenticated relays or via the Gmail API using OAuth2 requires a bit of work. In this article, I will provide sample code for sending rich HTML email with … Python: Sending HTML emails via Gmail API or SMTP relay
Zabbix distributes Docker images for each component. Not only does this mean you can quickly standup the monitoring solution, but upgrades also become a simple matter of trading up images. In this article, I will show how to stand up and then upgrade a zabbix installation using docker-compose.
A common task when configuring software is modifying a properties file to set or override behavior. And although properties files are simple key=value pairs, there are some corner cases that make it challenging. For example, when setting a key/value pair, the key may be commented out with a hash mark in front of the line. … Bash: setting and replacing values in a properties file use sed
The typical usage for xargs is feeding it multiple lines of input, and having it run commands using each line as an argument. But there are other cases when you have a single line separated by a delimiter or even a list of optionally quoted args, and I’ll show in this article how to handle … Bash: Running command on quoted list of parameters using xargs
Update Dec 2021: I have written an updated article for installing Docker on focal 20.04 Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization. These instructions are taken directly from the official Docker for Ubuntu page, but I wanted to reiterate those … Docker: Installing Docker CE on Ubuntu bionic 18.04
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. Building on my earlier article on the basics of introspection, I will now show how to inject custom … Python: Using a custom decorator to inspect function arguments
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. I will show a simple example of its usage in this article.
Using subprocess.Popen, subprocess.call, or subprocess.check_output will all invoke a process using Python, but if you want live output coming from stdout you need use subprocess.Popen in tandem with the Popen.poll method. In this article I will show how to invoke a process from Python and show stdout live without waiting for the process to complete. … Python: Getting live output from subprocess using poll
Especially when writing small Python utility programs, it can be tempting to use sys.argv to retrieve a small list of positional parameters from the command line. But small utilities turn into successful tools over time, and manually parsing optional parameters and switches will quickly overwhelm your code logic. In this article I will show how … Python: Parsing command line arguments with argparse
If you are using version 4+ of PowerShell, then instead of using the older makecert utility or the IIS Manager you can simply use New-SelfSignedCertificate cmdlet. In this article I’ll provide a small Powershell script that can assist in creating a self-signed certificate in the local machine personal store.
In a previous article I showed how to use a bridged network to give a VM access to the same network as the host, and then followed that up with creating a guest in a NAT network for network segmentation. In this article I will show how to create a network in Routed mode. This … KVM: Creating a guest VM on a network in routed mode
Iptables is a powerful utility for controlling network traffic coming through your host. But writing the rules that control the flow can be hard to troubleshoot when you have a complex network and multiple host interfaces. Luckily, you have the ability to insert log rules wherever necessary to get visibility into the packets flowing through … Ubuntu: Debug iptables by inserting a log rule
For troubleshooting deep rooted network issues, you may be accustomed to using tcpdump against a network interface on Linux. This same utility can be used on OpenWrt to troubleshoot network issues. The Opkg Package manager makes this easy to install.
In a previous article I showed how to use a bridged network to give a KVM Guest access to the same network as the Host. Now we will go over NAT networks which allow the KVM Guest to reach outside the network, but hosts from the outside cannot reach the guest directly. This opens up … KVM: Creating a guest VM on a NAT network
UPDATE September 2022: New article for bridged networks written for Ubuntu 22.04 In order to expose KVM virtual machines on the same network as your Host, you need to enable bridged networking. In this article, I’ll show how to implement KVM bridged networking on Ubuntu 18.04 bionic using Netplan. This bridged network will expose the … KVM: Creating a bridged network with NetPlan on Ubuntu 18.04 bionic
If you accidentally pushed a secret or password into a git repository, the BFG Repo-Clean utility is a convenient option for removing all traces of the secret from the entire git commit history. It is also possible to use ‘git-filter-branch‘, but I find BFG more convenient and faster.
If you are going to implement major changes to your publicly hosted WordPress site, it can be beneficial to create a local sandbox where those changes can be tested ahead of time. In this article, I will show how to use Docker Compose to create a local instance of WordPress, and the “All-in-One WP Migration” … WordPress: Cloning your WordPress site locally using Docker Compose
In a previous post, I described how to install and use the ‘govc‘ tool – which is a command line tool for working with VMware vCenter. When you do a listing on a virtual machine “govc ls -l -json <vmpath>”, you are returned back a very detailed JSON data structure that contains all the information … Python: JSONPath to extract vCenter information using govc
There are many tools that utilize json, and when it is relatively simple you can use standard modules or even custom coding to pull out the desired portions. But once these data structures reach a certain level of complexity you really should consider a Python module that implements JSONPath (analogous to xPath for XML). The … Python: Querying JSON files with JSONPath using jsonpath_rw_ext
The vSphere web GUI is a nice visual tool, but if you need to retrieve vCenter information in bulk or perform mass operations across VMs, then a command line tool such as govc in invaluable. govc is written in Go, which means it has support on Linux as well as most other platforms.
If you are transferring large files between systems, it can be advantageous to create a compressed archive using a tool like 7zip. Taking this a step further and splitting this archive into manageable chunks allows you to take advantage of parallel transfer. While there are many ways to do the same thing using utilities like … Linux: 7zip to split archives for use on Windows
If you have to ssh to a remote system to produce a block of json, it may be preceded and suffixed with login and system messages. Or you may have a program that sends json as the main part of its output, but also sends debug type messages to stdout that need to be cleaned. … Linux: sed to cleanup json that has errant text surrounding it
If you are running KVM on a console-only server, you still have the option to use the graphical virt-manager. You just need to specify the method of communication (ssh, tls, tcp, etc). In this article I will show how to use virt-manager from an Ubuntu client desktop to a server running KVM and libvirtd, with … KVM: virt-manager to connect to a remote console using qemu+ssh
Enabling NFS on a server and client allows the client to mount a remote filesystem. Below are the steps for an Ubuntu host server and Ubuntu remote client.
When troubleshooting an issue where ownership and permissions of a file or directories is questioned, we often go straight to chown/chmod and recursively set these values. However, this touches every file so if you are still in investigative mode it would be better to use “stat” to simply check the ownership and permissions first. If … Linux: Use stat to verify permissions and ownership
Updated Aug 2023: tested on Ubuntu 22 with QEMU 6.2.0/libvirt 8.0.0 and with Minikube v1.31.2 Minikube is a tool that runs a full Kubernetes stack on a host, making it ideal for local development and experimentation. We are taking it one step further by placing Minikube into a virtual machine run by the KVM virtualization … Kubernetes: running Minikube locally on Ubuntu using KVM
Updated April 2023: Tested on Ubuntu 22.04 LTS with X2GO sever 4.1.0 X2Go provides remote desktop access for Linux, similar to VNC or xRDP. It tunnels over ssh which can provide SSH public key authentication for security and is easily understood when opening firewall rules. Additionally, it is optimized for narrow bandwidth requirements, making it … Ubuntu: X2Go on Ubuntu bionic for remote desktop access
Especially with private git repositories that may be self-signed or have private CA, you may get the following error from the git client after a certificate has been updated: fatal: unable to access ‘https://git.mycompany.com/myuser/myrepo.git/’: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none This means that the git client cannot verify the integrity of the certificate … Git: client error, server certificate verification failed
If you receive this error upon trying to run commands from the CloudFoundry CLI: “Error read/writing config: unexpected end of JSON input” This is most likely due to an error in the “$HOME/.cf/config.json” file. Make sure the user has full ownership: ls -l ~/.cf/config.json chown $USER:$USER ~/.cf/config.json chmod 755 ~/.cf/config.json And if that does not … CloudFoundry: CLI error, unexpected end of JSON input
If you attempt an “apt-get update” and get a signature verification error from “dl.yarnpkg.com”, the problem is most likely an expired key. This can be resolved by reloading the new public key from the site: # add latest key curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add – # try refresh again sudo apt-get update Below … Ubuntu: apt-get error, yarn signature verification
BOSH is a project that unifies release, deployment, and lifecycle management of cloud based software. Software to be deployed via BOSH is called a release, and in this article I will use a very simple release to illustrate how to create, deploy, version, and revert these releases.
The AWS CLI provides a comprehensive command set, but if you want advanced functionality such as checking if an object already exists before you create it, or polling an instance until it is a certain state then you will need a slightly higher level of abstraction. My aws-common-funcs.sh script provides a set of Bash functions … AWS: Bash helper functions for common AWS CLI calls