Fabian

Bash: grep with LookBehind and LookAhead to isolate desired text


Categories: Scripting

grep has support for Perl compatible regular expressions (PCRE) by using the -P flag, and this provides a number of useful features.  In this article, I’ll show how LookBehind and LookAhead regular expression support can provide enhanced parsing abilities to your shell scripts. For example, consider an xml file “test.xml” with the contents: <root> <path>/my/data</path> … Bash: grep with LookBehind and LookAhead to isolate desired text

Ansible: regex capture groups with lineinfile to preserve yaml indentation


Categories: Scripting

One of the features of the ‘lineinfile‘ regexp parameter is the ability to use regular expression capture groups in the line output.  That allows you to extract values on a found line when constructing the output line. Specifically, that can mean pulling information such as hostname/port, file path, or preserving the yaml indentation of an … Ansible: regex capture groups with lineinfile to preserve yaml indentation

Ansible: lineinfile with regex to robustly populate key/value pairs in config file


Categories: Scripting

If your Ansible automation includes modifying an existing configuration file (versus managing your own fully templatized version of the config), then you will need to account for variations in that existing file when using ‘lineinfile‘ for key/value pairs. A naive lineinfile replacement will not find commented-out keys and might not even find valid keys that … Ansible: lineinfile with regex to robustly populate key/value pairs in config file

Bash: deep listing the most recently modified files in a directory


Categories: Scripting

Finding the most recently modified files in a directory can be extremely beneficial when you have been making changes in files throughout the directory structure as part of a work effort, and now need to go back and pinpoint everything that was changed. This command will provide you the 10 most recently modified files, excluding … Bash: deep listing the most recently modified files in a directory

Bash: Fixing an ASCII text file changed with Unicode character sequences


Categories: Linux

File encoding issues are difficult to diagnose and troubleshoot.  Most files in the operations world are expected to be text-only ASCII 7 bit, so if a file goes into UTF-8 encoding and has embedded Unicode character inserted, it can often throw off the tool chain or systems using the file. Here are two example files, … Bash: Fixing an ASCII text file changed with Unicode character sequences

CloudFoundry: Determining buildpack used by application


Categories: Containers

The “cf app” command will provide a brief expansion of a buildpack’s settings, but does not provide an exact buildpack name.  Luckily, this can easily be pulled using “cf curl” and the CloudFoundry API. Assuming you have the jq utility for parsing/querying json output: # set name of cloudfoundry app app=”my-cf-app” # using name, pull … CloudFoundry: Determining buildpack used by application

Python: Publishing and Consuming from RabbitMQ using Python


Categories: Python

The pika module for Python provides an easy interface for creating exchanges and queues as well as producers/consumers for RabbitMQ . In this article, I will provide examples of a producer and consumer written in Python3.  All source code is available on github.

Python: Examples of list comprehension


Categories: Python

Python list comprehension provides a concise and pythonic way of generating lists.  It allows you to select a subset based on a filter and apply a function to each element. Because although navigating every list entry, filtering certain records, and applying functions to specific fields is doable using basic ‘for’ statements, ‘if’ logic, and temporary … Python: Examples of list comprehension

RabbitMQ: Deleting a ghost queue that cannot be removed at the GUI/CLI


Categories: DevOps

If you get a timeout/errors trying to delete a RabbitMQ queue from  the management dashboard or CLI with an error similar in syntax to below: failed to perform operation on queue ” in vhost ” due to timeout Then you can attempt deletion using rabbitmqctl to evaluate an Erlang expression: rabbitmqctl eval ‘Q = {resource, … RabbitMQ: Deleting a ghost queue that cannot be removed at the GUI/CLI

Bash: Difference between two arrays


Categories: Linux, Scripting

Whether looking at differences in filenames, installed packages, etc. it can be useful to calculate the difference between two Bash arrays. SiegeX on stackoverflow.com offered the following function using awk, and I have built a full example available on github. function arraydiff() { awk ‘BEGIN{RS=ORS=” “} {NR==FNR?a[$0]++:a[$0]–} END{for(k in a)if(a[k])print k}’ <(echo -n “${!1}”) <(echo … Bash: Difference between two arrays

Ubuntu: Adding a root certificate authority


Categories: Linux

If your backend components or application servers use a custom CA (Certificate Authority), then you may need to add it to the system trusted root certificate store so that the standard tools and other utilities trust the TLS communication.

Bash: Examining each certificate in a yaml file using sed and openssl


Categories: Linux, Scripting

YAML is a popular syntax for configuration, and it is common to have certificate definitions embedded in these files. But since the cert is typically Base64 PEM encoded, it means you can’t easily view its attributes (subject, expiration date, etc) and so you are left with the manual task of copy-pasting it out, saving as … Bash: Examining each certificate in a yaml file using sed and openssl

Linux: Introducing latency and packet loss into network for testing


Categories: Linux

Within current distributions of Linux, there is a kernel component called netem that can be used to test and simulate the type of issues one would see over a Wide Area Network.  This component is managed with a tool called traffic controller. This can be helpful during testing/troubleshooting to emulate the network latency or packet … Linux: Introducing latency and packet loss into network for testing

KVM: Testing cloud-init locally using KVM for a CentOS cloud image


Categories: Linux, Virtualization

The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for a CentOS cloud image

KVM: Testing cloud-init locally using KVM for an Ubuntu cloud image


Categories: Linux, Virtualization

The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for an Ubuntu cloud image

KVM: Terraform and cloud-init to create local KVM resources


Categories: Linux, Virtualization

Terraform is a popular tool for provisioning infrastructure on cloud provider such as EC2 and Azure, but there is also a provider written for local KVM libvirt resources. Using the libvirt provider, we can use standard Terraform constructs to create local VMs, networks, and disks.

Bash: Associative array initialization and usage


Categories: Scripting

Just as in other programming languages, associative arrays in Bash are useful for search, set management, and keying into a list of values.  The label may be different, but whether called “map”, “dictionary”, or “associative array”, the same concepts apply. In its simplest form, here is an initialization and then loop through the key/value pairs. … Bash: Associative array initialization and usage

Bash: Appending to existing values using sed capture group


Categories: Scripting

sed is a powerful utility for transforming text.  One of the nice tricks with sed is the ability to reuse capture groups from the source string in the replacement value you are constructing. For example, if you have have the following kernel parameters in “/etc/default/grub” $ grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash” And wanted to append … Bash: Appending to existing values using sed capture group

Bash: Renaming files using shell parameter expansion


Categories: Scripting

Shell parameter expansion provides various ways to manipulate strings, and a convenient way to succinctly express renaming a set of files. In its simplest form, parameter expansion is simply ${parameter}.  But look at these examples: $ mystr=”TheQuickBrownFox.jpg” # chop off last 4 digits $ echo ${mystr:0:-4} TheQuickBrownFox # truncate end of string ‘.jpg’ $ echo … Bash: Renaming files using shell parameter expansion

GoLang: Go modules for package management during a multi-stage Docker build


Categories: Containers, Development

My previous article on multi-stage builds to create  Docker images for Go laid the foundation for using an intermediate image as the builder your Go binary.  However, this example was intentionally simplistic and did not address package and dependency management. Since the release of Go 1.11, the standard tooling has natively supported the concept of … GoLang: Go modules for package management during a multi-stage Docker build

GoLang: Using multi-stage builds to create clean Docker images


Categories: Containers, Development

The Go programming language is a natural fit for containers because it can compile down to a single statically-linked binary.  And if you place that single executable on top of scratch, a distroless image, or a small image like alpine, your final image has a minimal footprint which is great for consumption and reuse. But the … GoLang: Using multi-stage builds to create clean Docker images

GoLang: Installing the Go Programming language on Ubuntu


Categories: Development

The Go programming language has gotten considerable momentum, and the fact that it compiles down to a single statically linked binary has made it popular in containers, where a single executable binary fits the execution model perfectly. This article will detail installation on Ubuntu with the standard hello world validation.

Bash: Using shell or environment variables in awk output


Categories: Linux

If you are in the middle of a text processing pipeline, and need to insert a shell or environment variable into the output of awk, you can use the “-v” flag. Here are two files containing animal classifications: $ echo -e “shark=fish\ndolphin=mammal” > ocean.txt $ echo -e “dog=mammal\neagle=bird” > land.txt By passing the loop variable … Bash: Using shell or environment variables in awk output

Docker: Placing limits on cpu usage in containers


Categories: Containers, Linux

Containers themselves are light, but by default a container has access to all the CPU resources the Docker host kernel scheduler will allow. Internally Docker uses cgroups to limit CPU resources, and this is exposed as the flag “–cpus” when bringing up a docker container: sudo docker run -it –cpus=1.0 alpine:latest /bin/sh This will limit … Docker: Placing limits on cpu usage in containers

Docker: Placing limits on container memory using cgroups


Categories: Containers, Linux

Containers themselves are light, but by default a container has access to all the memory resources of the Docker host. Internally Docker uses cgroups to limit memory resources, and in its simplest form is exposed as the  flags “-m” and “–memory-swap” when bringing up a docker container. sudo docker run -it -m 8m –memory-swap 8m … Docker: Placing limits on container memory using cgroups

Docker: Use overlay2 with an xfs backing filesystem to limit rootfs size


Categories: Containers, Linux

If you are using the overlay2 storage driver, you can place limits on the rootfs within a container but only if using an xfs backing filesystem (not ext4). As a quick test of your Docker install, check your Docker storage driver and backing filesystem, then attempt to spin up a small alpine image with a … Docker: Use overlay2 with an xfs backing filesystem to limit rootfs size

Linux: Mounting a loopback ext4/xfs filesystem to isolate or enforce storage limits


Categories: Linux

The physical partitions and filesystem formats on your host are configured for your main workload, but if you want an application to use a specific filesystem (xfs, ext4, zfs) and size capacity without reconfiguration at the physical level then you can consider a loopback image. For example, if we create a 100Mb disk file named … Linux: Mounting a loopback ext4/xfs filesystem to isolate or enforce storage limits

Linux: Using xfs project quotas to limit capacity within a subdirectory


Categories: Linux

XFS is a journaled filesystem that has excellent parallel performance, and is licensed under the GPL which means it has been included in many Linux distributions. One of the features of XFS is the ability to enforce quotas based on user, group, and project.  In this article, I will show how to assign filesize quotas … Linux: Using xfs project quotas to limit capacity within a subdirectory