Fabian Lee : Software Engineer

PowerShell: Create Windows Scheduled Task to run Powershell script every hour

October 25, 2018
Categories: Scripting

If you are using a newer version of PowerShell, then by all means use the New-ScheduledTaskAction, New-ScheduledTaskTrigger, and Register-ScheduledTask and to create a Windows schedule task using PS scripting. But if you still need to be compatible back to PowerShell 2.0, and want to keep it simple, you can avoid using the Schedule.Service COM interface, … PowerShell: Create Windows Scheduled Task to run Powershell script every hour

KVM: Using dnsmasq for libvirt DNS resolution

October 22, 2018
Categories: Linux, Virtualization

On Linux host servers, libvirt uses a separate instance of dnsmasq for each virtual network. So if you want full name and reverse lookup for KVM guests on the default 192.168.122.0 network, you can configure this particular dnsmasq instance using virsh. The ability to customize the libvirt instances are limited, however, so if you need … KVM: Using dnsmasq for libvirt DNS resolution

Linux: Copy a directory preserving ownership, permissions, and modification date

October 15, 2018
Categories: Linux

File copying is about more than just content – the metadata for user ownership, permissions, and timestamps is often critical to retrieval and function. Below are the relevant switches for metadata preservation when using cp, rsync, and tar. These typically need sudo in order to work.

Ruby: Copying gems to hosts with limited internet access

October 14, 2018
Categories: Linux

In a production environment, it is common to have restricted internet access on production hosts. This means that using the standard ‘gem install’ and pulling gems from the rubygems.org repository directly on that host is not an option. What is needed is a way to use an internet-connected development host that matches the target production … Ruby: Copying gems to hosts with limited internet access

Ruby: Creating Selenium tests using headless Chrome and Ruby2

October 14, 2018
Categories: Scripting

Running Selenium tests is not just for quality assurance in development environments, it can also be used in production for baseline validation as well as performance and availability. If these tests are going to be run as part of a CI/CD pipeline or periodic monitoring check, then you will need to create automated headless jobs … Ruby: Creating Selenium tests using headless Chrome and Ruby2

Ubuntu: X11 forwarding to view GUI applications running on server hosts

October 14, 2018
Categories: Linux

Although server hosts typically have no graphical desktop and only serve console-based clients, these machines still have the ability to serve a GUI display screen to a remote desktop if necessary. The X11 protocol makes it possible to send the graphical display to a remote graphical desktop. Beyond the ability to run GUI utilities on … Ubuntu: X11 forwarding to view GUI applications running on server hosts

Linux: Excluding files based on extension and age with tar

October 12, 2018
Categories: Linux

If you are using tar to archive log files and find yourself needing to exclude certain extensions or older log files, tar can easily handle those type of situations. For example, to create a tar that excludes any rolled over logs that are zipped, use an ‘exclude’ like below: tar cvf mylogarchive.tar –exclude=*.gz –exclude=*.tgz * … Linux: Excluding files based on extension and age with tar

SaltStack: Escaping dollar signs in cmd.run parameters to avoid interpolation

October 10, 2018
Categories: Automation

The cmd.run state module is a handy way to run commands on a remote host. However, if your parameter values contain dollar signs ($) they will be interpolated in the target shell, which will lead to unexpected results. The trick is to escape properly by adding slashes before the dollar sign.

OpenWrt: Archive router configs for backup

October 10, 2018
Categories: OpenWrt

You should always have a backup of the main configuration settings of your OpenWrt device. A single change via LuCI or from the console can leave you with questions on what changed and how to get back to your original working condition. In this article, I’ll show you how to archive your main configuration settings, … OpenWrt: Archive router configs for backup

PuTTy: Bulk import PuTTy session definitions into the registry using Powershell

October 7, 2018
Categories: Python, Scripting

Putty is one of the first tools I install on any host or jumpbox. And creating a saved session definition is extremely helpful so I can get the right window size, scrollback, keep alives, color scheme, etc. but creating each session definition by hand is time consuming. In this article, I will show you how … PuTTy: Bulk import PuTTy session definitions into the registry using Powershell

KVM: Creating an Ubuntu VM with console-only access

October 6, 2018
Categories: Linux

If you are running KVM on a console-only host OS, you can still do a full Ubuntu installation on a KVM guest VM by using serial port output and the Ubuntu Server text-based installer. Once deployed, this guest VM can be accessed by using ssh or ‘virsh console’, without the need for a graphics display. … KVM: Creating an Ubuntu VM with console-only access

ELK: Deleting unassigned shards to restore cluster health

October 2, 2018
Categories: Logging

If your ElasticSearch cluster is not healthy because of unassigned shards, there are multiple resolution paths. This datadoghq article provides an excellent walk-through of how you can analyze and resolve the situation. The simplest case is when those unassigned shards are not required anymore, and deleting them restores cluster health. In this article, I will … ELK: Deleting unassigned shards to restore cluster health

Ubuntu: Customizing and repacking a deb file

September 28, 2018
Categories: Linux

Although there are utilities such as dpkg-deb for managing .deb packages, they can also be manipulated by the standard set of archival utilities: tar, ar, and gzip. This article will lead you through extracting the contents of a .deb file, making modifications to the installation scripts and default configuration files, then repackaging.

Linux: Excluding directories when using zip

September 26, 2018
Categories: Linux

If you are using zip and find yourself needing to exclude a directory (.git, build, etc), the “-x” exclude switch can provide that functionality. Take the following directory structure: $ find . . ./two.txt ./skipme ./skipme/three.txt ./one.txt You can exclude the entire ‘skipme’ folder and everything in it with: zip -r myzip.zip * -x skipme/*

KVM: Implementing linked clones with a backing file

September 24, 2018
Categories: Linux

If you need KVM to bring up a full guest OS as quickly as possible, consider preparing a base disk image that can be used as the basis for any number of linked clones by using a backing file. This article is related to a previous article, “Cloning and fixing an Ubuntu host using KVM“. … KVM: Implementing linked clones with a backing file

Ubuntu: Gathering hardware and BIOS details

September 23, 2018
Categories: Linux

Whether you need to flash the BIOS or check hardware compatibility for a virtualization engine, it is often necessary to gather details on your current hardware, BIOS, and CPU/DRAM feature set. In this article I’ll provide a starting list of commands you can use to gather this information.

Bash: Reading input from the console while looping over output of command

September 23, 2018
Categories: Linux, Scripting

If you are using the direct output of a command to feed a while loop in BASH, you may still want to take user input inside that loop. For example, if you need the user to confirm each line by pressing <ENTER>. Take a simple example like this: grep one test.txt | while read -r … Bash: Reading input from the console while looping over output of command

KVM: Deploying a nested version of VMware ESXi 6.7 inside KVM

September 19, 2018
Categories: Linux, Virtualization

Update Nov 2021: I have written a newer article that deploys ESXi 7.0U1. If you are running KVM on an Ubuntu server, you already have an excellent Type 1 virtualization engine. Luckily, if you need to test something specific to VMware you can always run ESXi 6.7 nested inside a KVM virtual machine. In this … KVM: Deploying a nested version of VMware ESXi 6.7 inside KVM

KVM: Cloning and fixing Ubuntu host using KVM

September 17, 2018
Categories: Linux, Virtualization

Usually the purpose of cloning a KVM guest OS is to have it use the same binaries and configuration as the parent, but allow it to run as an independent entity. When first cloned, the whole guest OS is cloned including host name, network settings, etc. and for those reasons you cannot run the parent … KVM: Cloning and fixing Ubuntu host using KVM

iptables: Running service as non-root, iptables to forward from privileged port

September 17, 2018
Categories: Linux

There are multiple approaches to allowing a process to run as a non-root user but still provide access to privileged ports (<1024). There are applications like Apache that handle this by starting the master process as root, and then worker processes as a less privileged user. Another way is setting the privilege on a binary … iptables: Running service as non-root, iptables to forward from privileged port

Ubuntu: Using iptables to forward tcp and udp requests

September 17, 2018
Categories: Linux

In previous articles I have described how to use a Squid proxy as a bridge to outside services such as http and ftp. Having hosts in your private networks use a single access point to services in the outside world institutes the type of control often required in production data centers. Having a single host … Ubuntu: Using iptables to forward tcp and udp requests

Vagrant: Fixing “error while downloading the metadata for this box”

September 16, 2018
Categories: Linux

If you see the message, “There was an error while downloading the metadata for this box”, with a 404 not found return message when doing a box update – make sure to check the URL listed in the Vagrant “metadata_url” files. For example, the “atlas.hashicorp.com” host has been deprecated in favor of “vagrantcloud.com” for some … Vagrant: Fixing “error while downloading the metadata for this box”

KVM: Alternate firmware BIOS for KVM

September 13, 2018
Categories: Linux

By default, KVM will use an older SeaBIOS x86 firmware for your virtual machines. If you want to use a more recent version of seaBIOS or want to drop the older BIOS standard and instead use the newer UUEFI specification (Unified Extensible Firmware Interface), KVM can support that with configuration changes. In this article, I … KVM: Alternate firmware BIOS for KVM

KVM: Building the latest SeaBIOS firmware for virtual machines

September 13, 2018
Categories: Linux

By default, KVM will use the SeaBIOS x86 firmware for virtual machines. This is typically installed as a dependent package which is years old. In this article, I will show how to built the latest SeaBIOS image from source.

KVM: Building the latest OVMF firmware for virtual machines

September 12, 2018
Categories: Linux

OVMF is an open-source project that implements the Unified Extensible Firmware Interface (UEFI) specification. UUEFI is designed to eventually replace the BIOS firmware interface. In this article, I will show how to build the latest OVMF image from source.

KVM: Bare metal virtualization on Ubuntu with KVM

August 27, 2018
Categories: Linux, Virtualization

Update September 2022: Validated these instructions on Ubuntu 22.04 If you are running an Ubuntu host, you have multiple choices for a virtualization hypervisor. I have written up several articles on using VirtualBox, but now let’s consider a bare metal hypervisor like KVM. KVM is a type 1 hypervisor implemented as a Linux kernel module … KVM: Bare metal virtualization on Ubuntu with KVM

SaltStack: salt-ssh for agentless automation on Ubuntu

July 1, 2018
Categories: Automation, DevOps

Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale. Even in the growing world of containerization, there is the need for bulk automation. This article will detail installation of Salt SSH which leverages the power of SaltStack without the requirements for an agent install.

SaltStack: Installing a Salt Master on Ubuntu Xenial

June 25, 2018
Categories: Automation, DevOps

Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale. Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion. This article will detail installation of the SaltStack master on Ubuntu Xenial 16.04, with validation using a single Minion. Note that … SaltStack: Installing a Salt Master on Ubuntu Xenial

Kubernetes: running Minikube locally on Ubuntu using VirtualBox

June 24, 2018
Categories: Containers

Updated article to latest Minikube, Feb 2019 Minikube is a tool that runs a Kubernetes stack inside a single VM being run by a local virtualization engine such as VirtualBox. This makes it ideal for local development and experimentation. In this article we’ll be going through installation and validation of a Minikube installation on Ubuntu … Kubernetes: running Minikube locally on Ubuntu using VirtualBox

Putty: Using pscp/plink for automated file transfer and command execution from a Windows host

June 23, 2018
Categories: Scripting

The Putty suite contains several executables beyond the main application. It also has pscp.exe, a command line scp client, and plink.exe which is a command line ssh client. These clients can be used to run file transfers and commands against hosts in console mode, perfect for automation of repetitive tasks.

WinSCP: Automated file transfer from a Windows host

June 23, 2018
Categories: Automation, Scripting

WinSCP is a Windows application for transferring files via fto or scp to remote host. A feature not many take advantage of is the ability to create automation scripts that can execute these transfers as silent batch jobs. In this article I will show how to use scripted FTP commands and SCP commands to automate … WinSCP: Automated file transfer from a Windows host

Docker: Using docker-compose and networking to link a Spring Boot app to an external service dependency

May 26, 2018
Categories: Containers

In earlier versions of Docker, links were used to connect two containers by enabling network access as well as sharing environment variables. This approach is being deprecated in favor of a shared network between the services. And instead of environment propagation, environment values will need to be explicitly added to each service that requires them. … Docker: Using docker-compose and networking to link a Spring Boot app to an external service dependency

Docker: Running a Spring Boot based app using docker-compose

May 24, 2018
Categories: Containers

Docker Compose gives us the ability to define services, ports, volumes, and networks in a single file. This file provides a convenient and unified view into what would otherwise be a long list of docker commands. In this article, you will run the Spring Boot based spring-music app in a Docker container using docker-compose.

Docker: Running a Spring Boot based app in a Docker container

May 24, 2018
Categories: Containers

Spring Boot is an easy way to create production grade Spring based web applications. In this article, I want to illustrate how easily a Spring Boot application can be deployed to run as a Docker container. Specifically, we will build the spring-music project into a single executable jar that will then be copied into a … Docker: Running a Spring Boot based app in a Docker container

Gradle: Upgrading the Gradle wrapper version

May 22, 2018
Categories: Development, Java

The recommended way to run a Gradle build is with the Gradle wrapper. You can check the current version of your wrapper with the command: $ ./gradlew –version If you are working with an older Gradle version and want to upgrade to take advantage of new features, you can use the ‘wrapper’ task specifying the … Gradle: Upgrading the Gradle wrapper version

Docker: Using docker-compose to link a MongoDB server and client

May 20, 2018
Categories: Containers, Linux

Docker Compose gives us the ability to define and orchestrate multiple containers in order to construct a service. In this article, we will use Docker Compose to create a MongoDB server and then another container that is used exclusively as a MongoDB client. While it is entirely possible to manually create these containers, links, and … Docker: Using docker-compose to link a MongoDB server and client

MongoDB: Installing a MongoDB client on Ubuntu

May 20, 2018
Categories: Development, Linux

In order to communicate with MongoDB using its default TCP protocol on port 27017, you will need a MongoDB client. There are many language bindings available, but in this article we’ll focus on the client available from the “mongodb-org-shell” Debian package.

Docker: Base image when deploying a GoLang binary in a container

May 10, 2018
Categories: Containers, Linux, Scripting

Update Oct 2020: multi-stage builds now provide a standard way to leverage a fat build image, while allowing your published image to remain small. This article is still useful for comparing base image sizes. GoLang applications are a great architectural fit for a Docker container because of their single binary executable. But you need to … Docker: Base image when deploying a GoLang binary in a container

GoLang: Installing the Go Programming language on Ubuntu 16.04

May 9, 2018
Categories: Development

Update: For the latest version of this article compatible with Ubuntu 22.04 and Go 1.19, see my newer article here. The Go programming language has gotten considerable momentum, and the fact that it compiles down to machine code has made it popular in containers like Docker where a single executable binary fits the execution model perfectly. This article … GoLang: Installing the Go Programming language on Ubuntu 16.04

Ubuntu: Creating an alias for common commands under bash

May 9, 2018
Categories: Linux

If you have Bash console commands that you type out frequently, consider creating an alias. By simply adding a “.bash_aliases” files in your home directory, you can easily define shortcuts to frequent commands. For example, if you frequently list all the files in a directory reverse sorted by date, you may end up typing “ls … Ubuntu: Creating an alias for common commands under bash