There are many Linux based ftp clients, but if you run into connectivity issues when using the HTTP CONNECT method through a proxy such as Squid, you may need to leverage more control by writing your interactions using a library such as ftp4j. In this article, I’ll provide a Java project with sample code that … Java: FTP with an HTTP proxy using the CONNECT method
With so many critical projects available in public git repositories like github, it is important to be able to contribute back additional functionality, tests, and documentation to the original project. While most projects do not allow a direct commit, contributions can be made by submitting a pull request to original repository. Often times, repositories are … Git: Contributing to a git project using a pull request
sudo provides users with temporary elevated privileges to perform operations. No matter what your security philosophy, sudo is more than likely enabled on your system if even for a limited number of users. And if it is enabled, creating an audit log of exactly what was run (and who ran it) is essential to reporting. … Ubuntu: Auditing sudo commands and forwarding audit logs using syslog
The mako template engine is a powerful module for Python used by many large sites including reddit.com. You simply push the variables you want into a context and then merge that with a text template. Additionally, mako allows you to take advantage of the full power of the Python language and libraries by allowing calls … Python: Calling python functions from mako templates
With the modern mantra of “everything is code”, operations and network teams must come to terms with how they want to work with source control in a team environment. Imagine a repository that contains configuration templates and scripts for maintaining an application or appliance. For a multi-member operations team who shares the responsibility for this … Git: Sharing a single git controlled folder among a group under Linux
In a production datacenter it would not be uncommon for internet access to be limited to domains whitelisted on a web proxy such as Squid. If this is the case, and you are using pip to install packages, then you will need to: Have your Squid administrators whitelist pypi.python.org Add the “–proxy” switch when invoking … Python: Using pip with a squid proxy
If you are in the vim camp, you may not appreciate git forcing you to emacs when committing. First, let me remind you how to save and then quit for emacs if you do get forced into that editor. CTRL-X CTRL-S (to save) then CTRL-X CTRL-C (to quit) And if you would like to avoid … Git: Forcing git to use vim for commit messages
If you are on an Ubuntu host and need to determine from which package a file originated, the ‘dpkg’ utility has this ability. For example, if you need to know which package the ‘mkpasswd’ binary came from: $ which mkpasswd /usr/bin/mkpasswd $ dpkg -S /usr/bin/mkpasswd whois: /usr/bin/mkpasswd Which tell us it came from the ‘whois’ … Ubuntu: Determining the package origin of a file
Update Nov 2021: I have written a newer article that deploys vCenter 7.0. If you have just virtualized the VMware ESXi 6.7 server on top of KVM, the next step will be to install vCenter 6.7 for its centralized control and additional feature set and management capabilities. In my last article we took KVM running … KVM: Deploy the VMware vCenter 6.7 appliance using the CLI installer
If you currently store sensitive credentials in plaintext to automate scripting or integration to other systems, you should consider an extra layer of security by storing them encrypted using GPG. There is no fullproof way to hide sensitive information for a service that also needs to decrypt them as part of normal operations (think DVD … Linux: Using GPG encrypted credentials for enhanced security
If you need to manipulate a Java archive such as a jar or war, and do not have access to the jar utility (it only comes packaged with the JDK, not JRE), you can always use the standard GNU zip/unzip utilities.
The sed utility is a powerful utility for doing text transformations. In this article, I will provide an example of how to insert a line before and after a match using sed, which is a common task for customizing configuration files. I have also written a related article on setting and replacing values in a … Linux: Using sed to insert lines before or after a match
If you are using a newer version of PowerShell, then by all means use the New-ScheduledTaskAction, New-ScheduledTaskTrigger, and Register-ScheduledTask and to create a Windows schedule task using PS scripting. But if you still need to be compatible back to PowerShell 2.0, and want to keep it simple, you can avoid using the Schedule.Service COM interface, … PowerShell: Create Windows Scheduled Task to run Powershell script every hour
On Linux host servers, libvirt uses a separate instance of dnsmasq for each virtual network. So if you want full name and reverse lookup for KVM guests on the default 192.168.122.0 network, you can configure this particular dnsmasq instance using virsh. The ability to customize the libvirt instances are limited, however, so if you need … KVM: Using dnsmasq for libvirt DNS resolution
File copying is about more than just content – the metadata for user ownership, permissions, and timestamps is often critical to retrieval and function. Below are the relevant switches for metadata preservation when using cp, rsync, and tar. These typically need sudo in order to work.
In a production environment, it is common to have restricted internet access on production hosts. This means that using the standard ‘gem install’ and pulling gems from the rubygems.org repository directly on that host is not an option. What is needed is a way to use an internet-connected development host that matches the target production … Ruby: Copying gems to hosts with limited internet access
Running Selenium tests is not just for quality assurance in development environments, it can also be used in production for baseline validation as well as performance and availability. If these tests are going to be run as part of a CI/CD pipeline or periodic monitoring check, then you will need to create automated headless jobs … Ruby: Creating Selenium tests using headless Chrome and Ruby2
Although server hosts typically have no graphical desktop and only serve console-based clients, these machines still have the ability to serve a GUI display screen to a remote desktop if necessary. The X11 protocol makes it possible to send the graphical display to a remote graphical desktop. Beyond the ability to run GUI utilities on … Ubuntu: X11 forwarding to view GUI applications running on server hosts
If you are using tar to archive log files and find yourself needing to exclude certain extensions or older log files, tar can easily handle those type of situations. For example, to create a tar that excludes any rolled over logs that are zipped, use an ‘exclude’ like below: tar cvf mylogarchive.tar –exclude=*.gz –exclude=*.tgz * … Linux: Excluding files based on extension and age with tar
The cmd.run state module is a handy way to run commands on a remote host. However, if your parameter values contain dollar signs ($) they will be interpolated in the target shell, which will lead to unexpected results. The trick is to escape properly by adding slashes before the dollar sign.
You should always have a backup of the main configuration settings of your OpenWrt device. A single change via LuCI or from the console can leave you with questions on what changed and how to get back to your original working condition. In this article, I’ll show you how to archive your main configuration settings, … OpenWrt: Archive router configs for backup
Putty is one of the first tools I install on any host or jumpbox. And creating a saved session definition is extremely helpful so I can get the right window size, scrollback, keep alives, color scheme, etc. but creating each session definition by hand is time consuming. In this article, I will show you how … PuTTy: Bulk import PuTTy session definitions into the registry using Powershell
If you are running KVM on a console-only host OS, you can still do a full Ubuntu installation on a KVM guest VM by using serial port output and the Ubuntu Server text-based installer. Once deployed, this guest VM can be accessed by using ssh or ‘virsh console’, without the need for a graphics display. … KVM: Creating an Ubuntu VM with console-only access
If your ElasticSearch cluster is not healthy because of unassigned shards, there are multiple resolution paths. This datadoghq article provides an excellent walk-through of how you can analyze and resolve the situation. The simplest case is when those unassigned shards are not required anymore, and deleting them restores cluster health. In this article, I will … ELK: Deleting unassigned shards to restore cluster health
Although there are utilities such as dpkg-deb for managing .deb packages, they can also be manipulated by the standard set of archival utilities: tar, ar, and gzip. This article will lead you through extracting the contents of a .deb file, making modifications to the installation scripts and default configuration files, then repackaging.
If you are using zip and find yourself needing to exclude a directory (.git, build, etc), the “-x” exclude switch can provide that functionality. Take the following directory structure: $ find . . ./two.txt ./skipme ./skipme/three.txt ./one.txt You can exclude the entire ‘skipme’ folder and everything in it with: zip -r myzip.zip * -x skipme/*
If you need KVM to bring up a full guest OS as quickly as possible, consider preparing a base disk image that can be used as the basis for any number of linked clones by using a backing file. This article is related to a previous article, “Cloning and fixing an Ubuntu host using KVM“. … KVM: Implementing linked clones with a backing file
Whether you need to flash the BIOS or check hardware compatibility for a virtualization engine, it is often necessary to gather details on your current hardware, BIOS, and CPU/DRAM feature set. In this article I’ll provide a starting list of commands you can use to gather this information.
If you are using the direct output of a command to feed a while loop in BASH, you may still want to take user input inside that loop. For example, if you need the user to confirm each line by pressing <ENTER>. Take a simple example like this: grep one test.txt | while read -r … Bash: Reading input from the console while looping over output of command
Update Nov 2021: I have written a newer article that deploys ESXi 7.0U1. If you are running KVM on an Ubuntu server, you already have an excellent Type 1 virtualization engine. Luckily, if you need to test something specific to VMware you can always run ESXi 6.7 nested inside a KVM virtual machine. In this … KVM: Deploying a nested version of VMware ESXi 6.7 inside KVM
Usually the purpose of cloning a KVM guest OS is to have it use the same binaries and configuration as the parent, but allow it to run as an independent entity. When first cloned, the whole guest OS is cloned including host name, network settings, etc. and for those reasons you cannot run the parent … KVM: Cloning and fixing Ubuntu host using KVM
There are multiple approaches to allowing a process to run as a non-root user but still provide access to privileged ports (<1024). There are applications like Apache that handle this by starting the master process as root, and then worker processes as a less privileged user. Another way is setting the privilege on a binary … iptables: Running service as non-root, iptables to forward from privileged port
In previous articles I have described how to use a Squid proxy as a bridge to outside services such as http and ftp. Having hosts in your private networks use a single access point to services in the outside world institutes the type of control often required in production data centers. Having a single host … Ubuntu: Using iptables to forward tcp and udp requests
If you see the message, “There was an error while downloading the metadata for this box”, with a 404 not found return message when doing a box update – make sure to check the URL listed in the Vagrant “metadata_url” files. For example, the “atlas.hashicorp.com” host has been deprecated in favor of “vagrantcloud.com” for some … Vagrant: Fixing “error while downloading the metadata for this box”
By default, KVM will use an older SeaBIOS x86 firmware for your virtual machines. If you want to use a more recent version of seaBIOS or want to drop the older BIOS standard and instead use the newer UUEFI specification (Unified Extensible Firmware Interface), KVM can support that with configuration changes. In this article, I … KVM: Alternate firmware BIOS for KVM
By default, KVM will use the SeaBIOS x86 firmware for virtual machines. This is typically installed as a dependent package which is years old. In this article, I will show how to built the latest SeaBIOS image from source.
OVMF is an open-source project that implements the Unified Extensible Firmware Interface (UEFI) specification. UUEFI is designed to eventually replace the BIOS firmware interface. In this article, I will show how to build the latest OVMF image from source.
Update September 2022: Validated these instructions on Ubuntu 22.04 If you are running an Ubuntu host, you have multiple choices for a virtualization hypervisor. I have written up several articles on using VirtualBox, but now let’s consider a bare metal hypervisor like KVM. KVM is a type 1 hypervisor implemented as a Linux kernel module … KVM: Bare metal virtualization on Ubuntu with KVM
Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale. Even in the growing world of containerization, there is the need for bulk automation. This article will detail installation of Salt SSH which leverages the power of SaltStack without the requirements for an agent install.
Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale. Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion. This article will detail installation of the SaltStack master on Ubuntu Xenial 16.04, with validation using a single Minion. Note that … SaltStack: Installing a Salt Master on Ubuntu Xenial