Terraform is a popular tool for provisioning infrastructure on cloud provider such as EC2 and Azure, but there is also a provider written for local KVM libvirt resources. Using the libvirt provider, we can use standard Terraform constructs to create local VMs, networks, and disks.
Volumes are the preferred method for persisting data for Docker containers. In this article, I will show how to create and use volumes for persistence, as well as tmpfs for temporary storage. At its absolute simplest, creating and mounting a volume backed by a local directory looks like: # make host directory mkdir -p /data … Docker: Working with local volumes and tmpfs mounts
If you are in the middle of a text processing pipeline, and need to insert a shell or environment variable into the output of awk, you can use the “-v” flag. Here are two files containing animal classifications: $ echo -e “shark=fish\ndolphin=mammal” > ocean.txt $ echo -e “dog=mammal\neagle=bird” > land.txt By passing the loop variable … Bash: Using shell or environment variables in awk output
Containers themselves are light, but by default a container has access to all the CPU resources the Docker host kernel scheduler will allow. Internally Docker uses cgroups to limit CPU resources, and this is exposed as the flag “–cpus” when bringing up a docker container: sudo docker run -it –cpus=1.0 alpine:latest /bin/sh This will limit … Docker: Placing limits on cpu usage in containers
Containers themselves are light, but by default a container has access to all the memory resources of the Docker host. Internally Docker uses cgroups to limit memory resources, and in its simplest form is exposed as the flags “-m” and “–memory-swap” when bringing up a docker container. sudo docker run -it -m 8m –memory-swap 8m … Docker: Placing limits on container memory using cgroups
Using head to get the first lines of a stream, and tail to get the last lines in a stream is intuitive. But if you need to skip the first few lines of a stream, then you use tail “-n +k” syntax. And to skip the last lines of a stream head “-n -k” syntax. … Bash: Skipping lines at the top or bottom of a stream
If you are using the overlay2 storage driver, you can place limits on the rootfs within a container but only if using an xfs backing filesystem (not ext4). As a quick test of your Docker install, check your Docker storage driver and backing filesystem, then attempt to spin up a small alpine image with a … Docker: Use overlay2 with an xfs backing filesystem to limit rootfs size
The physical partitions and filesystem formats on your host are configured for your main workload, but if you want an application to use a specific filesystem (xfs, ext4, zfs) and size capacity without reconfiguration at the physical level then you can consider a loopback image. For example, if we create a 100Mb disk file named … Linux: Mounting a loopback ext4/xfs filesystem to isolate or enforce storage limits
XFS is a journaled filesystem that has excellent parallel performance, and is licensed under the GPL which means it has been included in many Linux distributions. One of the features of XFS is the ability to enforce quotas based on user, group, and project. In this article, I will show how to assign filesize quotas … Linux: Using xfs project quotas to limit capacity within a subdirectory
This may come as a surprise, but Bash only supports integer arithmetic natively. If you need to perform calculations on floating point numbers, you will need to call out to a utility program like bc or Python. As a quick proof to yourself, try multiplying integers, then do the same with floating point numbers. $ … Bash: Performing floating arithmetic using bc
A common task when configuring software is modifying a properties file to set or override behavior. And although properties files are simple key=value pairs, there are some corner cases that make it challenging. For example, when setting a key/value pair, the key may be commented out with a hash mark in front of the line. … Bash: setting and replacing values in a properties file use sed
The typical usage for xargs is feeding it multiple lines of input, and having it run commands using each line as an argument. But there are other cases when you have a single line separated by a delimiter or even a list of optionally quoted args, and I’ll show in this article how to handle … Bash: Running command on quoted list of parameters using xargs
Update Dec 2021: I have written an updated article for installing Docker on focal 20.04 Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization. These instructions are taken directly from the official Docker for Ubuntu page, but I wanted to reiterate those … Docker: Installing Docker CE on Ubuntu bionic 18.04
In a previous article I showed how to use a bridged network to give a VM access to the same network as the host, and then followed that up with creating a guest in a NAT network for network segmentation. In this article I will show how to create a network in Routed mode. This … KVM: Creating a guest VM on a network in routed mode
Iptables is a powerful utility for controlling network traffic coming through your host. But writing the rules that control the flow can be hard to troubleshoot when you have a complex network and multiple host interfaces. Luckily, you have the ability to insert log rules wherever necessary to get visibility into the packets flowing through … Ubuntu: Debug iptables by inserting a log rule
In a previous article I showed how to use a bridged network to give a KVM Guest access to the same network as the Host. Now we will go over NAT networks which allow the KVM Guest to reach outside the network, but hosts from the outside cannot reach the guest directly. This opens up … KVM: Creating a guest VM on a NAT network
UPDATE September 2022: New article for bridged networks written for Ubuntu 22.04 In order to expose KVM virtual machines on the same network as your Host, you need to enable bridged networking. In this article, I’ll show how to implement KVM bridged networking on Ubuntu 18.04 bionic using Netplan. This bridged network will expose the … KVM: Creating a bridged network with NetPlan on Ubuntu 18.04 bionic
The vSphere web GUI is a nice visual tool, but if you need to retrieve vCenter information in bulk or perform mass operations across VMs, then a command line tool such as govc in invaluable. govc is written in Go, which means it has support on Linux as well as most other platforms.
If you are transferring large files between systems, it can be advantageous to create a compressed archive using a tool like 7zip. Taking this a step further and splitting this archive into manageable chunks allows you to take advantage of parallel transfer. While there are many ways to do the same thing using utilities like … Linux: 7zip to split archives for use on Windows
If you have to ssh to a remote system to produce a block of json, it may be preceded and suffixed with login and system messages. Or you may have a program that sends json as the main part of its output, but also sends debug type messages to stdout that need to be cleaned. … Linux: sed to cleanup json that has errant text surrounding it
If you are running KVM on a console-only server, you still have the option to use the graphical virt-manager. You just need to specify the method of communication (ssh, tls, tcp, etc). In this article I will show how to use virt-manager from an Ubuntu client desktop to a server running KVM and libvirtd, with … KVM: virt-manager to connect to a remote console using qemu+ssh
Enabling NFS on a server and client allows the client to mount a remote filesystem. Below are the steps for an Ubuntu host server and Ubuntu remote client.
When troubleshooting an issue where ownership and permissions of a file or directories is questioned, we often go straight to chown/chmod and recursively set these values. However, this touches every file so if you are still in investigative mode it would be better to use “stat” to simply check the ownership and permissions first. If … Linux: Use stat to verify permissions and ownership
Updated April 2023: Tested on Ubuntu 22.04 LTS with X2GO sever 4.1.0 X2Go provides remote desktop access for Linux, similar to VNC or xRDP. It tunnels over ssh which can provide SSH public key authentication for security and is easily understood when opening firewall rules. Additionally, it is optimized for narrow bandwidth requirements, making it … Ubuntu: X2Go on Ubuntu bionic for remote desktop access
Especially with private git repositories that may be self-signed or have private CA, you may get the following error from the git client after a certificate has been updated: fatal: unable to access ‘https://git.mycompany.com/myuser/myrepo.git/’: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none This means that the git client cannot verify the integrity of the certificate … Git: client error, server certificate verification failed
If you attempt an “apt-get update” and get a signature verification error from “dl.yarnpkg.com”, the problem is most likely an expired key. This can be resolved by reloading the new public key from the site: # add latest key curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add – # try refresh again sudo apt-get update Below … Ubuntu: apt-get error, yarn signature verification
sudo provides users with temporary elevated privileges to perform operations. No matter what your security philosophy, sudo is more than likely enabled on your system if even for a limited number of users. And if it is enabled, creating an audit log of exactly what was run (and who ran it) is essential to reporting. … Ubuntu: Auditing sudo commands and forwarding audit logs using syslog
If you are on an Ubuntu host and need to determine from which package a file originated, the ‘dpkg’ utility has this ability. For example, if you need to know which package the ‘mkpasswd’ binary came from: $ which mkpasswd /usr/bin/mkpasswd $ dpkg -S /usr/bin/mkpasswd whois: /usr/bin/mkpasswd Which tell us it came from the ‘whois’ … Ubuntu: Determining the package origin of a file
Update Nov 2021: I have written a newer article that deploys vCenter 7.0. If you have just virtualized the VMware ESXi 6.7 server on top of KVM, the next step will be to install vCenter 6.7 for its centralized control and additional feature set and management capabilities. In my last article we took KVM running … KVM: Deploy the VMware vCenter 6.7 appliance using the CLI installer
If you currently store sensitive credentials in plaintext to automate scripting or integration to other systems, you should consider an extra layer of security by storing them encrypted using GPG. There is no fullproof way to hide sensitive information for a service that also needs to decrypt them as part of normal operations (think DVD … Linux: Using GPG encrypted credentials for enhanced security
If you need to manipulate a Java archive such as a jar or war, and do not have access to the jar utility (it only comes packaged with the JDK, not JRE), you can always use the standard GNU zip/unzip utilities.
The sed utility is a powerful utility for doing text transformations. In this article, I will provide an example of how to insert a line before and after a match using sed, which is a common task for customizing configuration files. I have also written a related article on setting and replacing values in a … Linux: Using sed to insert lines before or after a match
On Linux host servers, libvirt uses a separate instance of dnsmasq for each virtual network. So if you want full name and reverse lookup for KVM guests on the default 192.168.122.0 network, you can configure this particular dnsmasq instance using virsh. The ability to customize the libvirt instances are limited, however, so if you need … KVM: Using dnsmasq for libvirt DNS resolution
File copying is about more than just content – the metadata for user ownership, permissions, and timestamps is often critical to retrieval and function. Below are the relevant switches for metadata preservation when using cp, rsync, and tar. These typically need sudo in order to work.
In a production environment, it is common to have restricted internet access on production hosts. This means that using the standard ‘gem install’ and pulling gems from the rubygems.org repository directly on that host is not an option. What is needed is a way to use an internet-connected development host that matches the target production … Ruby: Copying gems to hosts with limited internet access
Although server hosts typically have no graphical desktop and only serve console-based clients, these machines still have the ability to serve a GUI display screen to a remote desktop if necessary. The X11 protocol makes it possible to send the graphical display to a remote graphical desktop. Beyond the ability to run GUI utilities on … Ubuntu: X11 forwarding to view GUI applications running on server hosts
If you are using tar to archive log files and find yourself needing to exclude certain extensions or older log files, tar can easily handle those type of situations. For example, to create a tar that excludes any rolled over logs that are zipped, use an ‘exclude’ like below: tar cvf mylogarchive.tar –exclude=*.gz –exclude=*.tgz * … Linux: Excluding files based on extension and age with tar
If you are running KVM on a console-only host OS, you can still do a full Ubuntu installation on a KVM guest VM by using serial port output and the Ubuntu Server text-based installer. Once deployed, this guest VM can be accessed by using ssh or ‘virsh console’, without the need for a graphics display. … KVM: Creating an Ubuntu VM with console-only access
Although there are utilities such as dpkg-deb for managing .deb packages, they can also be manipulated by the standard set of archival utilities: tar, ar, and gzip. This article will lead you through extracting the contents of a .deb file, making modifications to the installation scripts and default configuration files, then repackaging.
If you are using zip and find yourself needing to exclude a directory (.git, build, etc), the “-x” exclude switch can provide that functionality. Take the following directory structure: $ find . . ./two.txt ./skipme ./skipme/three.txt ./one.txt You can exclude the entire ‘skipme’ folder and everything in it with: zip -r myzip.zip * -x skipme/*