The pika module for Python provides an easy interface for creating exchanges and queues as well as producers/consumers for RabbitMQ . In this article, I will provide examples of a producer and consumer written in Python3. All source code is available on github.
Python list comprehension provides a concise and pythonic way of generating lists. It allows you to select a subset based on a filter and apply a function to each element. Because although navigating every list entry, filtering certain records, and applying functions to specific fields is doable using basic ‘for’ statements, ‘if’ logic, and temporary … Python: Examples of list comprehension
If you get a timeout/errors trying to delete a RabbitMQ queue from the management dashboard or CLI with an error similar in syntax to below: failed to perform operation on queue ” in vhost ” due to timeout Then you can attempt deletion using rabbitmqctl to evaluate an Erlang expression: rabbitmqctl eval ‘Q = {resource, … RabbitMQ: Deleting a ghost queue that cannot be removed at the GUI/CLI
If you need to output only lines before (or after) a unique line identified by a regular expression, then here are examples of using sed. All examples in this article can be found in github. Consider the following lines of content in the variable “$lines” $ echo “$lines” ape swings frog jumps logs dog barks … Bash: output all lines before/after line identified by regex
Whether looking at differences in filenames, installed packages, etc. it can be useful to calculate the difference between two Bash arrays. SiegeX on stackoverflow.com offered the following function using awk, and I have built a full example available on github. function arraydiff() { awk ‘BEGIN{RS=ORS=” “} {NR==FNR?a[$0]++:a[$0]–} END{for(k in a)if(a[k])print k}’ <(echo -n “${!1}”) <(echo … Bash: Difference between two arrays
If your backend components or application servers use a custom CA (Certificate Authority), then you may need to add it to the system trusted root certificate store so that the standard tools and other utilities trust the TLS communication.
YAML is a popular syntax for configuration, and it is common to have certificate definitions embedded in these files. But since the cert is typically Base64 PEM encoded, it means you can’t easily view its attributes (subject, expiration date, etc) and so you are left with the manual task of copy-pasting it out, saving as … Bash: Examining each certificate in a yaml file using sed and openssl
The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for a RHEL cloud image
Within current distributions of Linux, there is a kernel component called netem that can be used to test and simulate the type of issues one would see over a Wide Area Network. This component is managed with a tool called traffic controller. This can be helpful during testing/troubleshooting to emulate the network latency or packet … Linux: Introducing latency and packet loss into network for testing
The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for a CentOS cloud image
The ability to quickly stand up a guest OS with cloud-init is most often associated with deployment of virtual machines in an IaaS like EC2 or Azure. But cloud-init is not just for remote cloud providers, and using cloud-init for local images that can be quickly deployed in KVM works great for local development and … KVM: Testing cloud-init locally using KVM for an Ubuntu cloud image
Terraform is a popular tool for provisioning infrastructure on cloud provider such as EC2 and Azure, but there is also a provider written for local KVM libvirt resources. Using the libvirt provider, we can use standard Terraform constructs to create local VMs, networks, and disks.
Just as in other programming languages, associative arrays in Bash are useful for search, set management, and keying into a list of values. The label may be different, but whether called “map”, “dictionary”, or “associative array”, the same concepts apply. In its simplest form, here is an initialization and then loop through the key/value pairs. … Bash: Associative array initialization and usage
sed is a powerful utility for transforming text. One of the nice tricks with sed is the ability to reuse capture groups from the source string in the replacement value you are constructing. For example, if you have have the following kernel parameters in “/etc/default/grub” $ grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash” And wanted to append … Bash: Appending to existing values using sed capture group
The =~ binary operator provides the ability to compare a string to a POSIX extended regular expression in the shell. Take note, the right-hand side regex cannot be surrounded by quotes or it will be treated as a regular string, it cannot contain spaces, and must conform to POSIX regex rules and use character classes … Bash: Using BASH_REMATCH to pull capture groups from a regex
Shell parameter expansion provides various ways to manipulate strings, and a convenient way to succinctly express renaming a set of files. In its simplest form, parameter expansion is simply ${parameter}. But look at these examples: $ mystr=”TheQuickBrownFox.jpg” # chop off last 4 digits $ echo ${mystr:0:-4} TheQuickBrownFox # truncate end of string ‘.jpg’ $ echo … Bash: Renaming files using shell parameter expansion
My previous article on multi-stage builds to create Docker images for Go laid the foundation for using an intermediate image as the builder your Go binary. However, this example was intentionally simplistic and did not address package and dependency management. Since the release of Go 1.11, the standard tooling has natively supported the concept of … GoLang: Go modules for package management during a multi-stage Docker build
The Go programming language is a natural fit for containers because it can compile down to a single statically-linked binary. And if you place that single executable on top of scratch, a distroless image, or a small image like alpine, your final image has a minimal footprint which is great for consumption and reuse. But the … GoLang: Using multi-stage builds to create clean Docker images
The Go programming language has gotten considerable momentum, and the fact that it compiles down to a single statically linked binary has made it popular in containers, where a single executable binary fits the execution model perfectly. This article will detail installation on Ubuntu with the standard hello world validation.
Volumes are the preferred method for persisting data for Docker containers. In this article, I will show how to create and use volumes for persistence, as well as tmpfs for temporary storage. At its absolute simplest, creating and mounting a volume backed by a local directory looks like: # make host directory mkdir -p /data … Docker: Working with local volumes and tmpfs mounts
If you are in the middle of a text processing pipeline, and need to insert a shell or environment variable into the output of awk, you can use the “-v” flag. Here are two files containing animal classifications: $ echo -e “shark=fish\ndolphin=mammal” > ocean.txt $ echo -e “dog=mammal\neagle=bird” > land.txt By passing the loop variable … Bash: Using shell or environment variables in awk output
Containers themselves are light, but by default a container has access to all the CPU resources the Docker host kernel scheduler will allow. Internally Docker uses cgroups to limit CPU resources, and this is exposed as the flag “–cpus” when bringing up a docker container: sudo docker run -it –cpus=1.0 alpine:latest /bin/sh This will limit … Docker: Placing limits on cpu usage in containers
Containers themselves are light, but by default a container has access to all the memory resources of the Docker host. Internally Docker uses cgroups to limit memory resources, and in its simplest form is exposed as the flags “-m” and “–memory-swap” when bringing up a docker container. sudo docker run -it -m 8m –memory-swap 8m … Docker: Placing limits on container memory using cgroups
Using head to get the first lines of a stream, and tail to get the last lines in a stream is intuitive. But if you need to skip the first few lines of a stream, then you use tail “-n +k” syntax. And to skip the last lines of a stream head “-n -k” syntax. … Bash: Skipping lines at the top or bottom of a stream
awk is a powerful utility for text processing, but the command itself is surrounded by single quotes and unfortunately does not allow you to escape single quotes within the output. To workaround this, you can represent the single quote as its hexidecimal escape code “\x27”. $ echo -e “name=Ben\nname=Jerry” | awk -F’=’ ‘{ printf(“insert into … Linux: Outputting single quotes in awk output
If you are using the overlay2 storage driver, you can place limits on the rootfs within a container but only if using an xfs backing filesystem (not ext4). As a quick test of your Docker install, check your Docker storage driver and backing filesystem, then attempt to spin up a small alpine image with a … Docker: Use overlay2 with an xfs backing filesystem to limit rootfs size
The physical partitions and filesystem formats on your host are configured for your main workload, but if you want an application to use a specific filesystem (xfs, ext4, zfs) and size capacity without reconfiguration at the physical level then you can consider a loopback image. For example, if we create a 100Mb disk file named … Linux: Mounting a loopback ext4/xfs filesystem to isolate or enforce storage limits
XFS is a journaled filesystem that has excellent parallel performance, and is licensed under the GPL which means it has been included in many Linux distributions. One of the features of XFS is the ability to enforce quotas based on user, group, and project. In this article, I will show how to assign filesize quotas … Linux: Using xfs project quotas to limit capacity within a subdirectory
If you are looking for basic HTTP file uploading using Flask, you can use the code in my python-flask-upload-files project on github as an example. This project exercises multiple scenarios: Upload a single file in chunked mode (not forced to save entirety to disk) Upload a single file Upload multiple files Error message shown when … Python: Using Flask to upload files
If you need to call attention to a message within your script, a splash of color is often the best way to get noticed in the sea of monochrome text. The standard ANSI escape codes are supported at most consoles, and in its simplest form can be used like this: RED=’\033[0;31m’ GREEN=’\033[0;32m’ NC=’\033[0m’ echo -e … Bash: Outputting text in color for readability
This may come as a surprise, but Bash only supports integer arithmetic natively. If you need to perform calculations on floating point numbers, you will need to call out to a utility program like bc or Python. As a quick proof to yourself, try multiplying integers, then do the same with floating point numbers. $ … Bash: Performing floating arithmetic using bc
If you are using Flask to generate dynamic content of significant size, such as large binary images/pdf or large text-based datasets, then you need to consider streaming to minimize the memory footprint of Flask and preserve scalability. Using an inner generate function and a yield allows Flask to return chunks of data back to the … Python: Using Flask to stream chunked dynamic content to end users
Update Jan 2022: If you would rather have a mail server running in Kubernetes, see my article here. There are many reasons you might want your own Postfix server: maybe you need to test various relay settings, or validate the ability of your mail script to use TLS, or maybe you are a developer that … Docker: Running a Postfix container for testing mail during development
Although sending a plain text message through an unauthenticated SMTP relay might be easily done with a few lines of Python code, sending rich HTML email through authenticated relays or via the Gmail API using OAuth2 requires a bit of work. In this article, I will provide sample code for sending rich HTML email with … Python: Sending HTML emails via Gmail API or SMTP relay
Zabbix distributes Docker images for each component. Not only does this mean you can quickly standup the monitoring solution, but upgrades also become a simple matter of trading up images. In this article, I will show how to stand up and then upgrade a zabbix installation using docker-compose.
A common task when configuring software is modifying a properties file to set or override behavior. And although properties files are simple key=value pairs, there are some corner cases that make it challenging. For example, when setting a key/value pair, the key may be commented out with a hash mark in front of the line. … Bash: setting and replacing values in a properties file use sed
The typical usage for xargs is feeding it multiple lines of input, and having it run commands using each line as an argument. But there are other cases when you have a single line separated by a delimiter or even a list of optionally quoted args, and I’ll show in this article how to handle … Bash: Running command on quoted list of parameters using xargs
Update Dec 2021: I have written an updated article for installing Docker on focal 20.04 Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization. These instructions are taken directly from the official Docker for Ubuntu page, but I wanted to reiterate those … Docker: Installing Docker CE on Ubuntu bionic 18.04
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. Building on my earlier article on the basics of introspection, I will now show how to inject custom … Python: Using a custom decorator to inspect function arguments
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. I will show a simple example of its usage in this article.