curl has a “write-out” ability to display information after a completed transfer that can be used to support Bash scripting. In it’s simplest form, a curl request looks like this: page=”https://www.google.com” curl $page But this leaves a lot to be desired in a scripting scenario: curl does not exit with a non-zero code on failure, … Bash: Capturing HTTP status code using curl write-out
If you have multiple values coming through the Bash input pipeline, it can be difficult to process these into a complex, formatted set of arguments unless you use intermediate temporary files. But one way to neatly put together a complex set of arguments from an input pipeline with multiple values is to use awk printf … Bash: using multiple values from an input pipeline to construct and execute a command
It is not uncommon to find an installation page where you are asked to download a packaged release from the github site. But even if you are using the “latest” alias to grab the most recent, you may still need to know the exact release tag for unzipping, configuration management, etc. Luckily, a simple query … Bash: Determining latest github release tag and version
Although the GCP console provides a nice interface for displaying which user/service account is in which IAM security role (IAM & Admin > IAM), it can be difficult to analyze using gcloud get-iam-policy because of the inner array of ‘members’ returned. However, if you use the flattening ability of gcloud, it becomes much easier to … GCP: Analyzing members of IAM role using gcloud filtering and jq
grep has support for Perl compatible regular expressions (PCRE) by using the -P flag, and this provides a number of useful features. In this article, I’ll show how LookBehind and LookAhead regular expression support can provide enhanced parsing abilities to your shell scripts. For example, consider an xml file “test.xml” with the contents: <root> <path>/my/data</path> … Bash: grep with LookBehind and LookAhead to isolate desired text
One of the features of the ‘lineinfile‘ regexp parameter is the ability to use regular expression capture groups in the line output. That allows you to extract values on a found line when constructing the output line. Specifically, that can mean pulling information such as hostname/port, file path, or preserving the yaml indentation of an … Ansible: regex capture groups with lineinfile to preserve yaml indentation
If your Ansible automation includes modifying an existing configuration file (versus managing your own fully templatized version of the config), then you will need to account for variations in that existing file when using ‘lineinfile‘ for key/value pairs. A naive lineinfile replacement will not find commented-out keys and might not even find valid keys that … Ansible: lineinfile with regex to robustly populate key/value pairs in config file
Finding the most recently modified files in a directory can be extremely beneficial when you have been making changes in files throughout the directory structure as part of a work effort, and now need to go back and pinpoint everything that was changed. This command will provide you the 10 most recently modified files, excluding … Bash: deep listing the most recently modified files in a directory
If you need to output only lines before (or after) a unique line identified by a regular expression, then here are examples of using sed. All examples in this article can be found in github. Consider the following lines of content in the variable “$lines” $ echo “$lines” ape swings frog jumps logs dog barks … Bash: output all lines before/after line identified by regex
Whether looking at differences in filenames, installed packages, etc. it can be useful to calculate the difference between two Bash arrays. SiegeX on stackoverflow.com offered the following function using awk, and I have built a full example available on github. function arraydiff() { awk ‘BEGIN{RS=ORS=” “} {NR==FNR?a[$0]++:a[$0]–} END{for(k in a)if(a[k])print k}’ <(echo -n “${!1}”) <(echo … Bash: Difference between two arrays
YAML is a popular syntax for configuration, and it is common to have certificate definitions embedded in these files. But since the cert is typically Base64 PEM encoded, it means you can’t easily view its attributes (subject, expiration date, etc) and so you are left with the manual task of copy-pasting it out, saving as … Bash: Examining each certificate in a yaml file using sed and openssl
Just as in other programming languages, associative arrays in Bash are useful for search, set management, and keying into a list of values. The label may be different, but whether called “map”, “dictionary”, or “associative array”, the same concepts apply. In its simplest form, here is an initialization and then loop through the key/value pairs. … Bash: Associative array initialization and usage
sed is a powerful utility for transforming text. One of the nice tricks with sed is the ability to reuse capture groups from the source string in the replacement value you are constructing. For example, if you have have the following kernel parameters in “/etc/default/grub” $ grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash” And wanted to append … Bash: Appending to existing values using sed capture group
The =~ binary operator provides the ability to compare a string to a POSIX extended regular expression in the shell. Take note, the right-hand side regex cannot be surrounded by quotes or it will be treated as a regular string, it cannot contain spaces, and must conform to POSIX regex rules and use character classes … Bash: Using BASH_REMATCH to pull capture groups from a regex
Shell parameter expansion provides various ways to manipulate strings, and a convenient way to succinctly express renaming a set of files. In its simplest form, parameter expansion is simply ${parameter}. But look at these examples: $ mystr=”TheQuickBrownFox.jpg” # chop off last 4 digits $ echo ${mystr:0:-4} TheQuickBrownFox # truncate end of string ‘.jpg’ $ echo … Bash: Renaming files using shell parameter expansion
awk is a powerful utility for text processing, but the command itself is surrounded by single quotes and unfortunately does not allow you to escape single quotes within the output. To workaround this, you can represent the single quote as its hexidecimal escape code “\x27”. $ echo -e “name=Ben\nname=Jerry” | awk -F’=’ ‘{ printf(“insert into … Linux: Outputting single quotes in awk output
If you need to call attention to a message within your script, a splash of color is often the best way to get noticed in the sea of monochrome text. The standard ANSI escape codes are supported at most consoles, and in its simplest form can be used like this: RED=’\033[0;31m’ GREEN=’\033[0;32m’ NC=’\033[0m’ echo -e … Bash: Outputting text in color for readability
This may come as a surprise, but Bash only supports integer arithmetic natively. If you need to perform calculations on floating point numbers, you will need to call out to a utility program like bc or Python. As a quick proof to yourself, try multiplying integers, then do the same with floating point numbers. $ … Bash: Performing floating arithmetic using bc
The typical usage for xargs is feeding it multiple lines of input, and having it run commands using each line as an argument. But there are other cases when you have a single line separated by a delimiter or even a list of optionally quoted args, and I’ll show in this article how to handle … Bash: Running command on quoted list of parameters using xargs
Especially when writing small Python utility programs, it can be tempting to use sys.argv to retrieve a small list of positional parameters from the command line. But small utilities turn into successful tools over time, and manually parsing optional parameters and switches will quickly overwhelm your code logic. In this article I will show how … Python: Parsing command line arguments with argparse
If you are using version 4+ of PowerShell, then instead of using the older makecert utility or the IIS Manager you can simply use New-SelfSignedCertificate cmdlet. In this article I’ll provide a small Powershell script that can assist in creating a self-signed certificate in the local machine personal store.
The AWS CLI provides a comprehensive command set, but if you want advanced functionality such as checking if an object already exists before you create it, or polling an instance until it is a certain state then you will need a slightly higher level of abstraction. My aws-common-funcs.sh script provides a set of Bash functions … AWS: Bash helper functions for common AWS CLI calls
If you are using a newer version of PowerShell, then by all means use the New-ScheduledTaskAction, New-ScheduledTaskTrigger, and Register-ScheduledTask and to create a Windows schedule task using PS scripting. But if you still need to be compatible back to PowerShell 2.0, and want to keep it simple, you can avoid using the Schedule.Service COM interface, … PowerShell: Create Windows Scheduled Task to run Powershell script every hour
Running Selenium tests is not just for quality assurance in development environments, it can also be used in production for baseline validation as well as performance and availability. If these tests are going to be run as part of a CI/CD pipeline or periodic monitoring check, then you will need to create automated headless jobs … Ruby: Creating Selenium tests using headless Chrome and Ruby2
Putty is one of the first tools I install on any host or jumpbox. And creating a saved session definition is extremely helpful so I can get the right window size, scrollback, keep alives, color scheme, etc. but creating each session definition by hand is time consuming. In this article, I will show you how … PuTTy: Bulk import PuTTy session definitions into the registry using Powershell
If you are using the direct output of a command to feed a while loop in BASH, you may still want to take user input inside that loop. For example, if you need the user to confirm each line by pressing <ENTER>. Take a simple example like this: grep one test.txt | while read -r … Bash: Reading input from the console while looping over output of command
The Putty suite contains several executables beyond the main application. It also has pscp.exe, a command line scp client, and plink.exe which is a command line ssh client. These clients can be used to run file transfers and commands against hosts in console mode, perfect for automation of repetitive tasks.
WinSCP is a Windows application for transferring files via fto or scp to remote host. A feature not many take advantage of is the ability to create automation scripts that can execute these transfers as silent batch jobs. In this article I will show how to use scripted FTP commands and SCP commands to automate … WinSCP: Automated file transfer from a Windows host
Update Oct 2020: multi-stage builds now provide a standard way to leverage a fat build image, while allowing your published image to remain small. This article is still useful for comparing base image sizes. GoLang applications are a great architectural fit for a Docker container because of their single binary executable. But you need to … Docker: Base image when deploying a GoLang binary in a container
SMTP mail relays exposed to the internet typically use a combination of SSL and authenticated SMTP to avoid abuse by malicious actors. This is an excellent choice from a security perspective, but makes smoke testing a bit more complex than just opening telnet.
Zabbix low-level discovery (LLD) provides a way to create an array of related items, triggers, or graphs without needing to know the exact number of entities up front. The easiest way to populate the keys of a discovery item is to add a “UserParameter” in zabbix_agentd.conf, and then the Zabbix agent will invokes a script which returns the set … Zabbix: LLD low-level discovery returning multiple values
The Docker console commands for listing and viewing containers and images (ps, images, history, inspect) provides a wealth of information, but when you are managing hundreds of containers, a graph view of the container inventory and their dependencies can be critical for operations. Dockviz can help you visualize your containers and images by creating an PNG image … Docker: Visualizing image hierarchy and container dependency using dockviz
Update Oct 2022: This article has now been written for GoLang 1.19 on Ubuntu 22.04. Go has changed the way it handles SIGURG signals, and Systemd services no longer directly forward to syslog. Read my newer article here. The Go language with its simplicity, concurrency support, rich package ecosystem, and ability to compile down to a single … GoLang: Running a Go binary as a systemd service on Ubuntu 16.04
ElasticSearch very often serves as a repository for monitoring, logging, and business data. As such, integrations with external system are a requirement. The Go programming language with its convenient deployment binary and rich set of packages can easily serve as a bridge between these systems and the ElasticSearch server. We will use the olivere/elastic package for this purpose, it is … ELK: Connecting to ElasticSearch with a Go client
The Go language with its simplicity, concurrency support, rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu. However, the Go language does not natively provide a reliable way to daemonize itself. In this article I will describe how to take a couple of simple Go language programs, … GoLang: Running a Go binary as a SysV service on Ubuntu 14.04
The open-source Zabbix monitoring solution has a published, simple binary protocol that allows you to send metrics to the Zabbix server without relying on the Zabbix Agent – which makes it very convenient for integration with other parts of your infrastructure. In this article, I’ll show how to use the go-zabbix package for sending metrics to … Zabbix: Sending Zabbix metrics using a Go client
Downloading 3rd party packages from github is made very simple in the Go language with the import statement. But similar to other languages, the complexity of versions and inter-dependencies begs the use of a package manager for any projects that are non-trivial (think npm for Javascript, pip for Python, Maven for Java, etc.). Glide is a package manager for the Go … GoLang: Glide for Go language package management
The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems. This fosters development of community-driven modules like Ryan Day’s zabbix Go language package, which is an easy way to automate Zabbix tasks like creating hosts and manipulating other back end structures. One of the nice things … Zabbix: Zabbix REST API using a Go client
Using 3rd party packages from github is made very simple in the Go language with the import statement. But one problem is that “go get” will always pull the HEAD of the master branch and there is no way to explicitly specify another branch. The ultimate answer would be to use a package dependency manager like Glide, which … GoLang: Vendor directory for github branches other than master
A nice feature of the Go language is the ability to build binaries for multiple platforms directly from a single source system. As an example, even from a development Windows 7 32-bit machine, you can build binaries for both 64 bit Linux and Windows 2012 Servers. Before Go 1.5, you needed a compiler for the target architecture, … GoLang: Cross Compiling for Linux and Windows platforms