Using subprocess.Popen, subprocess.call, or subprocess.check_output will all invoke a process using Python, but if you want live output coming from stdout you need use subprocess.Popen in tandem with the Popen.poll method. In this article I will show how to invoke a process from Python and show stdout live without waiting for the process to complete. … Python: Getting live output from subprocess using poll
Especially when writing small Python utility programs, it can be tempting to use sys.argv to retrieve a small list of positional parameters from the command line. But small utilities turn into successful tools over time, and manually parsing optional parameters and switches will quickly overwhelm your code logic. In this article I will show how … Python: Parsing command line arguments with argparse
If you are using version 4+ of PowerShell, then instead of using the older makecert utility or the IIS Manager you can simply use New-SelfSignedCertificate cmdlet. In this article I’ll provide a small Powershell script that can assist in creating a self-signed certificate in the local machine personal store.
In a previous article I showed how to use a bridged network to give a VM access to the same network as the host, and then followed that up with creating a guest in a NAT network for network segmentation. In this article I will show how to create a network in Routed mode. This … KVM: Creating a guest VM on a network in routed mode
Iptables is a powerful utility for controlling network traffic coming through your host. But writing the rules that control the flow can be hard to troubleshoot when you have a complex network and multiple host interfaces. Luckily, you have the ability to insert log rules wherever necessary to get visibility into the packets flowing through … Ubuntu: Debug iptables by inserting a log rule
For troubleshooting deep rooted network issues, you may be accustomed to using tcpdump against a network interface on Linux. This same utility can be used on OpenWrt to troubleshoot network issues. The Opkg Package manager makes this easy to install.
In a previous article I showed how to use a bridged network to give a KVM Guest access to the same network as the Host. Now we will go over NAT networks which allow the KVM Guest to reach outside the network, but hosts from the outside cannot reach the guest directly. This opens up … KVM: Creating a guest VM on a NAT network
UPDATE September 2022: New article for bridged networks written for Ubuntu 22.04 In order to expose KVM virtual machines on the same network as your Host, you need to enable bridged networking. In this article, I’ll show how to implement KVM bridged networking on Ubuntu 18.04 bionic using Netplan. This bridged network will expose the … KVM: Creating a bridged network with NetPlan on Ubuntu 18.04 bionic
If you accidentally pushed a secret or password into a git repository, the BFG Repo-Clean utility is a convenient option for removing all traces of the secret from the entire git commit history. It is also possible to use ‘git-filter-branch‘, but I find BFG more convenient and faster.
If you are going to implement major changes to your publicly hosted WordPress site, it can be beneficial to create a local sandbox where those changes can be tested ahead of time. In this article, I will show how to use Docker Compose to create a local instance of WordPress, and the “All-in-One WP Migration” … WordPress: Cloning your WordPress site locally using Docker Compose
In a previous post, I described how to install and use the ‘govc‘ tool – which is a command line tool for working with VMware vCenter. When you do a listing on a virtual machine “govc ls -l -json <vmpath>”, you are returned back a very detailed JSON data structure that contains all the information … Python: JSONPath to extract vCenter information using govc
There are many tools that utilize json, and when it is relatively simple you can use standard modules or even custom coding to pull out the desired portions. But once these data structures reach a certain level of complexity you really should consider a Python module that implements JSONPath (analogous to xPath for XML). The … Python: Querying JSON files with JSONPath using jsonpath_rw_ext
The vSphere web GUI is a nice visual tool, but if you need to retrieve vCenter information in bulk or perform mass operations across VMs, then a command line tool such as govc in invaluable. govc is written in Go, which means it has support on Linux as well as most other platforms.
If you are transferring large files between systems, it can be advantageous to create a compressed archive using a tool like 7zip. Taking this a step further and splitting this archive into manageable chunks allows you to take advantage of parallel transfer. While there are many ways to do the same thing using utilities like … Linux: 7zip to split archives for use on Windows
If you have to ssh to a remote system to produce a block of json, it may be preceded and suffixed with login and system messages. Or you may have a program that sends json as the main part of its output, but also sends debug type messages to stdout that need to be cleaned. … Linux: sed to cleanup json that has errant text surrounding it
If you are running KVM on a console-only server, you still have the option to use the graphical virt-manager. You just need to specify the method of communication (ssh, tls, tcp, etc). In this article I will show how to use virt-manager from an Ubuntu client desktop to a server running KVM and libvirtd, with … KVM: virt-manager to connect to a remote console using qemu+ssh
Enabling NFS on a server and client allows the client to mount a remote filesystem. Below are the steps for an Ubuntu host server and Ubuntu remote client.
When troubleshooting an issue where ownership and permissions of a file or directories is questioned, we often go straight to chown/chmod and recursively set these values. However, this touches every file so if you are still in investigative mode it would be better to use “stat” to simply check the ownership and permissions first. If … Linux: Use stat to verify permissions and ownership
Updated Aug 2023: tested on Ubuntu 22 with QEMU 6.2.0/libvirt 8.0.0 and with Minikube v1.31.2 Minikube is a tool that runs a full Kubernetes stack on a host, making it ideal for local development and experimentation. We are taking it one step further by placing Minikube into a virtual machine run by the KVM virtualization … Kubernetes: running Minikube locally on Ubuntu using KVM
Updated April 2023: Tested on Ubuntu 22.04 LTS with X2GO sever 4.1.0 X2Go provides remote desktop access for Linux, similar to VNC or xRDP. It tunnels over ssh which can provide SSH public key authentication for security and is easily understood when opening firewall rules. Additionally, it is optimized for narrow bandwidth requirements, making it … Ubuntu: X2Go on Ubuntu bionic for remote desktop access
Especially with private git repositories that may be self-signed or have private CA, you may get the following error from the git client after a certificate has been updated: fatal: unable to access ‘https://git.mycompany.com/myuser/myrepo.git/’: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none This means that the git client cannot verify the integrity of the certificate … Git: client error, server certificate verification failed
If you receive this error upon trying to run commands from the CloudFoundry CLI: “Error read/writing config: unexpected end of JSON input” This is most likely due to an error in the “$HOME/.cf/config.json” file. Make sure the user has full ownership: ls -l ~/.cf/config.json chown $USER:$USER ~/.cf/config.json chmod 755 ~/.cf/config.json And if that does not … CloudFoundry: CLI error, unexpected end of JSON input
If you attempt an “apt-get update” and get a signature verification error from “dl.yarnpkg.com”, the problem is most likely an expired key. This can be resolved by reloading the new public key from the site: # add latest key curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add – # try refresh again sudo apt-get update Below … Ubuntu: apt-get error, yarn signature verification
BOSH is a project that unifies release, deployment, and lifecycle management of cloud based software. Software to be deployed via BOSH is called a release, and in this article I will use a very simple release to illustrate how to create, deploy, version, and revert these releases.
The AWS CLI provides a comprehensive command set, but if you want advanced functionality such as checking if an object already exists before you create it, or polling an instance until it is a certain state then you will need a slightly higher level of abstraction. My aws-common-funcs.sh script provides a set of Bash functions … AWS: Bash helper functions for common AWS CLI calls
BOSH is a project that unifies release, deployment, and lifecycle management of cloud based software. It also serves as the underlying infrastructure for deploying the CloudFoundry PaaS. In this article, I will lead you through deploying the BOSH Director to Amazon EC2 which is the first step in deploying both CloudFoundry CFAR as well as CFCR … CloudFoundry: Installing a BOSH Director on AWS
Amazon EC2 provides a web interface for managing IaaS, but for repeatable infrastructure deployment what you really want is the ability to deploy and manage this infrastructure using an API or command line tool. The AWS SDK for Python (Boto3) provides a lower level as well as resource level API for managing and creating infrastructure. This … AWS: Installing the AWS SDK for Python on Ubuntu
Amazon EC2 provides a web interface for managing IaaS, but for repeatable infrastructure deployment what you really want is the ability to deploy and manage this infrastructure using an API or command line tool. In this article we want to focus on the CLI (command line interface), which shields us from the API innards and … AWS: Installing the AWS CLI on Ubuntu
There are many Linux based ftp clients, but if you run into connectivity issues when using the HTTP CONNECT method through a proxy such as Squid, you may need to leverage more control by writing your interactions using a library such as ftp4j. In this article, I’ll provide a Java project with sample code that … Java: FTP with an HTTP proxy using the CONNECT method
With so many critical projects available in public git repositories like github, it is important to be able to contribute back additional functionality, tests, and documentation to the original project. While most projects do not allow a direct commit, contributions can be made by submitting a pull request to original repository. Often times, repositories are … Git: Contributing to a git project using a pull request
sudo provides users with temporary elevated privileges to perform operations. No matter what your security philosophy, sudo is more than likely enabled on your system if even for a limited number of users. And if it is enabled, creating an audit log of exactly what was run (and who ran it) is essential to reporting. … Ubuntu: Auditing sudo commands and forwarding audit logs using syslog
The mako template engine is a powerful module for Python used by many large sites including reddit.com. You simply push the variables you want into a context and then merge that with a text template. Additionally, mako allows you to take advantage of the full power of the Python language and libraries by allowing calls … Python: Calling python functions from mako templates
With the modern mantra of “everything is code”, operations and network teams must come to terms with how they want to work with source control in a team environment. Imagine a repository that contains configuration templates and scripts for maintaining an application or appliance. For a multi-member operations team who shares the responsibility for this … Git: Sharing a single git controlled folder among a group under Linux
In a production datacenter it would not be uncommon for internet access to be limited to domains whitelisted on a web proxy such as Squid. If this is the case, and you are using pip to install packages, then you will need to: Have your Squid administrators whitelist pypi.python.org Add the “–proxy” switch when invoking … Python: Using pip with a squid proxy
If you are in the vim camp, you may not appreciate git forcing you to emacs when committing. First, let me remind you how to save and then quit for emacs if you do get forced into that editor. CTRL-X CTRL-S (to save) then CTRL-X CTRL-C (to quit) And if you would like to avoid … Git: Forcing git to use vim for commit messages
If you are on an Ubuntu host and need to determine from which package a file originated, the ‘dpkg’ utility has this ability. For example, if you need to know which package the ‘mkpasswd’ binary came from: $ which mkpasswd /usr/bin/mkpasswd $ dpkg -S /usr/bin/mkpasswd whois: /usr/bin/mkpasswd Which tell us it came from the ‘whois’ … Ubuntu: Determining the package origin of a file
Update Nov 2021: I have written a newer article that deploys vCenter 7.0. If you have just virtualized the VMware ESXi 6.7 server on top of KVM, the next step will be to install vCenter 6.7 for its centralized control and additional feature set and management capabilities. In my last article we took KVM running … KVM: Deploy the VMware vCenter 6.7 appliance using the CLI installer
If you currently store sensitive credentials in plaintext to automate scripting or integration to other systems, you should consider an extra layer of security by storing them encrypted using GPG. There is no fullproof way to hide sensitive information for a service that also needs to decrypt them as part of normal operations (think DVD … Linux: Using GPG encrypted credentials for enhanced security
If you need to manipulate a Java archive such as a jar or war, and do not have access to the jar utility (it only comes packaged with the JDK, not JRE), you can always use the standard GNU zip/unzip utilities.
The sed utility is a powerful utility for doing text transformations. In this article, I will provide an example of how to insert a line before and after a match using sed, which is a common task for customizing configuration files. I have also written a related article on setting and replacing values in a … Linux: Using sed to insert lines before or after a match