If you need to bootstrap a GCP project’s infrastructure, one of the first things you will want is a service account. The creation of the service account, creating its key, and then assigning binding roles can all be done from the GCP console but for scripting purposes can also be done using the gcloud utility. … GCP: Using gcloud to create and configure a service account
Although the GCP console provides a nice interface for displaying which user/service account is in which IAM security role (IAM & Admin > IAM), it can be difficult to analyze using gcloud get-iam-policy because of the inner array of ‘members’ returned. However, if you use the flattening ability of gcloud, it becomes much easier to … GCP: Analyzing members of IAM role using gcloud filtering and jq
Update July 2021: I have seen errors with external snapshots of volumes on versions of QEMU/KVM/libvirt from Ubuntu 20 Focal. Adding note on using internal snapshot on volume backed by qcow2. Internal snapshots created on QEMU copy-on-write (qcow2) disks are the most commonly used snapshot when using libvirt. It is easy to see why; … KVM: creating and reverting libvirt external snapshots
grep has support for Perl compatible regular expressions (PCRE) by using the -P flag, and this provides a number of useful features. In this article, I’ll show how LookBehind and LookAhead regular expression support can provide enhanced parsing abilities to your shell scripts. For example, consider an xml file “test.xml” with the contents: <root> <path>/my/data</path> … Bash: grep with LookBehind and LookAhead to isolate desired text
A centralized authentication/authorization mechanism is a key part of a security stance as well as a convenience to end users. There are multiple packages and systems to achieve this, and in this article I will focus on integrating back into Windows Active Directory using SSSD for login and group membership. When complete, you will have … Ansible: Login to Ubuntu with Windows Active Directory using SSSD
One of the features of the ‘lineinfile‘ regexp parameter is the ability to use regular expression capture groups in the line output. That allows you to extract values on a found line when constructing the output line. Specifically, that can mean pulling information such as hostname/port, file path, or preserving the yaml indentation of an … Ansible: regex capture groups with lineinfile to preserve yaml indentation
If your Ansible automation includes modifying an existing configuration file (versus managing your own fully templatized version of the config), then you will need to account for variations in that existing file when using ‘lineinfile‘ for key/value pairs. A naive lineinfile replacement will not find commented-out keys and might not even find valid keys that … Ansible: lineinfile with regex to robustly populate key/value pairs in config file
Finding the most recently modified files in a directory can be extremely beneficial when you have been making changes in files throughout the directory structure as part of a work effort, and now need to go back and pinpoint everything that was changed. This command will provide you the 10 most recently modified files, excluding … Bash: deep listing the most recently modified files in a directory
Even where there is an existing git pull request (PR) addressing a bug fix or feature you might find critical for your use case, it is not uncommon for it to be awaiting a merge by the project owner. As an example, the hellofresh company has done a great service to the community offering up … Git: Incorporating multiple pull requests from the main project into your fork
Because of the inherited nature of .gitignore, it can be tricky to determine if you have any files in your repository folder that are “hiding” from source control without your knowledge. This may be in the form of files that should be under source control but got caught in a broad inherited .gitignore rule, or … Git: Identifying files that .gitignore is purposely skipping
File encoding issues are difficult to diagnose and troubleshoot. Most files in the operations world are expected to be text-only ASCII 7 bit, so if a file goes into UTF-8 encoding and has embedded Unicode character inserted, it can often throw off the tool chain or systems using the file. Here are two example files, … Bash: Fixing an ASCII text file changed with Unicode character sequences
In a production environment where a proxy such as squid is necessary to reach the public internet, you may need use apt-add-repository to install a package from a ppa. Using the Ansible ppa as an example, the command from the documentation is: sudo apt-add-repository –yes –update ppa:ansible/ansible You might be tempted to believe that if … Ubuntu: Using add-apt-repository with a proxy
Desktop screen sharing is ubiquitous, but there is still a lot of value in sharing a live terminal screen with another set of users that can not only view, but also have the ability to copy-paste and even provide input. The tmux utility can create a session using a shared socket. To start a tmux … Bash: Sharing a terminal screen among users with tmux
The “cf app” command will provide a brief expansion of a buildpack’s settings, but does not provide an exact buildpack name. Luckily, this can easily be pulled using “cf curl” and the CloudFoundry API. Assuming you have the jq utility for parsing/querying json output: # set name of cloudfoundry app app=”my-cf-app” # using name, pull … CloudFoundry: Determining buildpack used by application
Bash has a rich set of control structures but it is still common to see logic expressions used a shorthand for if-then-else/tertiary statements. This can provide a curt and single-line ingestable way of controlling the flow of your Bash script. For example: $ [ 1 -eq 1 ] && echo “correct, 1 does indeed equal … Bash: Using logic expressions as a shorthand for if-then-else control
The pika module for Python provides an easy interface for creating exchanges and queues as well as producers/consumers for RabbitMQ . In this article, I will provide examples of a producer and consumer written in Python3. All source code is available on github.
Python list comprehension provides a concise and pythonic way of generating lists. It allows you to select a subset based on a filter and apply a function to each element. Because although navigating every list entry, filtering certain records, and applying functions to specific fields is doable using basic ‘for’ statements, ‘if’ logic, and temporary … Python: Examples of list comprehension
If you get a timeout/errors trying to delete a RabbitMQ queue from the management dashboard or CLI with an error similar in syntax to below: failed to perform operation on queue ” in vhost ” due to timeout Then you can attempt deletion using rabbitmqctl to evaluate an Erlang expression: rabbitmqctl eval ‘Q = {resource, … RabbitMQ: Deleting a ghost queue that cannot be removed at the GUI/CLI
If you need to output only lines before (or after) a unique line identified by a regular expression, then here are examples of using sed. All examples in this article can be found in github. Consider the following lines of content in the variable “$lines” $ echo “$lines” ape swings frog jumps logs dog barks … Bash: output all lines before/after line identified by regex
Whether looking at differences in filenames, installed packages, etc. it can be useful to calculate the difference between two Bash arrays. SiegeX on stackoverflow.com offered the following function using awk, and I have built a full example available on github. function arraydiff() { awk ‘BEGIN{RS=ORS=” “} {NR==FNR?a[$0]++:a[$0]–} END{for(k in a)if(a[k])print k}’ <(echo -n “${!1}”) <(echo … Bash: Difference between two arrays
If your backend components or application servers use a custom CA (Certificate Authority), then you may need to add it to the system trusted root certificate store so that the standard tools and other utilities trust the TLS communication.
YAML is a popular syntax for configuration, and it is common to have certificate definitions embedded in these files. But since the cert is typically Base64 PEM encoded, it means you can’t easily view its attributes (subject, expiration date, etc) and so you are left with the manual task of copy-pasting it out, saving as … Bash: Examining each certificate in a yaml file using sed and openssl
The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for a RHEL cloud image
Within current distributions of Linux, there is a kernel component called netem that can be used to test and simulate the type of issues one would see over a Wide Area Network. This component is managed with a tool called traffic controller. This can be helpful during testing/troubleshooting to emulate the network latency or packet … Linux: Introducing latency and packet loss into network for testing
The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for a CentOS cloud image
The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for an Ubuntu cloud image
Terraform is a popular tool for provisioning infrastructure on cloud provider such as EC2 and Azure, but there is also a provider written for local KVM libvirt resources. Using the libvirt provider, we can use standard Terraform constructs to create local VMs, networks, and disks.
Just as in other programming languages, associative arrays in Bash are useful for search, set management, and keying into a list of values. The label may be different, but whether called “map”, “dictionary”, or “associative array”, the same concepts apply. In its simplest form, here is an initialization and then loop through the key/value pairs. … Bash: Associative array initialization and usage
sed is a powerful utility for transforming text. One of the nice tricks with sed is the ability to reuse capture groups from the source string in the replacement value you are constructing. For example, if you have have the following kernel parameters in “/etc/default/grub” $ grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash” And wanted to append … Bash: Appending to existing values using sed capture group
The =~ binary operator provides the ability to compare a string to a POSIX extended regular expression in the shell. Take note, the right-hand side regex cannot be surrounded by quotes or it will be treated as a regular string, it cannot contain spaces, and must conform to POSIX regex rules and use character classes … Bash: Using BASH_REMATCH to pull capture groups from a regex
Shell parameter expansion provides various ways to manipulate strings, and a convenient way to succinctly express renaming a set of files. In its simplest form, parameter expansion is simply ${parameter}. But look at these examples: $ mystr=”TheQuickBrownFox.jpg” # chop off last 4 digits $ echo ${mystr:0:-4} TheQuickBrownFox # truncate end of string ‘.jpg’ $ echo … Bash: Renaming files using shell parameter expansion
My previous article on multi-stage builds to create Docker images for Go laid the foundation for using an intermediate image as the builder your Go binary. However, this example was intentionally simplistic and did not address package and dependency management. Since the release of Go 1.11, the standard tooling has natively supported the concept of … GoLang: Go modules for package management during a multi-stage Docker build
The Go programming language is a natural fit for containers because it can compile down to a single statically-linked binary. And if you place that single executable on top of scratch, a distroless image, or a small image like alpine, your final image has a minimal footprint which is great for consumption and reuse. But the … GoLang: Using multi-stage builds to create clean Docker images
The Go programming language has gotten considerable momentum, and the fact that it compiles down to a single statically linked binary has made it popular in containers, where a single executable binary fits the execution model perfectly. This article will detail installation on Ubuntu with the standard hello world validation.
Volumes are the preferred method for persisting data for Docker containers. In this article, I will show how to create and use volumes for persistence, as well as tmpfs for temporary storage. At its absolute simplest, creating and mounting a volume backed by a local directory looks like: # make host directory mkdir -p /data … Docker: Working with local volumes and tmpfs mounts
If you are in the middle of a text processing pipeline, and need to insert a shell or environment variable into the output of awk, you can use the “-v” flag. Here are two files containing animal classifications: $ echo -e “shark=fish\ndolphin=mammal” > ocean.txt $ echo -e “dog=mammal\neagle=bird” > land.txt By passing the loop variable … Bash: Using shell or environment variables in awk output
Containers themselves are light, but by default a container has access to all the CPU resources the Docker host kernel scheduler will allow. Internally Docker uses cgroups to limit CPU resources, and this is exposed as the flag “–cpus” when bringing up a docker container: sudo docker run -it –cpus=1.0 alpine:latest /bin/sh This will limit … Docker: Placing limits on cpu usage in containers
Containers themselves are light, but by default a container has access to all the memory resources of the Docker host. Internally Docker uses cgroups to limit memory resources, and in its simplest form is exposed as the flags “-m” and “–memory-swap” when bringing up a docker container. sudo docker run -it -m 8m –memory-swap 8m … Docker: Placing limits on container memory using cgroups
Using head to get the first lines of a stream, and tail to get the last lines in a stream is intuitive. But if you need to skip the first few lines of a stream, then you use tail “-n +k” syntax. And to skip the last lines of a stream head “-n -k” syntax. … Bash: Skipping lines at the top or bottom of a stream
awk is a powerful utility for text processing, but the command itself is surrounded by single quotes and unfortunately does not allow you to escape single quotes within the output. To workaround this, you can represent the single quote as its hexidecimal escape code “\x27”. $ echo -e “name=Ben\nname=Jerry” | awk -F’=’ ‘{ printf(“insert into … Linux: Outputting single quotes in awk output