The Kubernetes Downward API allows a pod to get access to metadata about itself and the cluster without creating a tight coupling to the Kubernetes API. For example, information such as pod name, labels, annotations, IP address, node, and cpu/memory limits can be made available inside the pod. In this article, I’ll show how to … Kubernetes: using the Downward API to access pod/container metadata
In a previous article I discussed the advantages to keeping container images in the private Google Container Registry of a project. And if you have a GKE cluster in the exact same project, then image pulls happen seamlessly without any additional configuration required. However, if the GKE cluster is in a different project than the … GCP: pulling an image from the Container Registry of another project
Docker hub now enforces pull rate limits (since November 2020). And unfortunately, this limit is often reached at critical moments such as upgrades or infrastructure events when bulk pod recreation is happening. One way to avoid this problem is to place your images into an alternate image registry. This could mean a lot of work … GCP: pushing GKE images into gcr.io to avoid pull rate limits
If you are doing a substitution with sed but need to exclude a specific line or pattern, that can be accomplished by prefixing an exclusion and using “!”. For example, to substitute “hello” for “goodbye” in the following content, but to skip any line containing “world”, use syntax like the following: $ sed “/world/! s/hello/goodbye/g” … Bash: sed substitution with an exclusion pattern
If you need to determine the version of the nginx ingress controller deployed, then you can invoke the ingress controller binary with the ‘–version’ flag. But this binary is located in the ingress-nginx-controller pod, so do a ‘kubectl exec’ like below. # show all running nginx ingress pods kubectl get pods -n $ingress_ns -l app.kubernetes.io/name=ingress-nginx … Kubernetes: detecting the installed version of nginx ingress
Pulling the most recent script/file/asset from a remote host may be preferable, but you would like to provide a fallback to a local file just in case the remote pull fails. This is relatively easy to do with Ansible by allowing the remote pull to ‘ignore_errors’ and then doing a local file copy if an … Ansible: preferring a pull from a URL with fallback to a local file
If you are executing gkectl commands that scale or modify your worker nodes and hit a problem, the first place to go is into the gkectl logs. But if you need to dig deeper there are a couple of CRD that can assist in troubleshooting. First, make sure you have executed gkectl with high verbosity … GCP: troubleshooting nodepool replica changes for Anthos on-premise
Once you introduce an istio sidecar proxy into your deployment, it becomes another point at which you might need to troubleshoot network connectivity to the primary container. Assuming you have deployed a pod with an app label “helloworld” in the default namespace listening on port 5000, you can use a command like the following to … Kubernetes: testing pod communication directly from istio sidecar proxy
If your istio ingress Gateway is in a different namespace than your VirtualService, then you need to make sure you prefix the gateway reference with that namespace. For example, if your istio ingress Gateway is in the ‘default’ namespace, yet your Deployment, Service, and VirtualService are in the namespace ‘helloworld’. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: … Kubernetes: istio Gateway in a different namespace than VirtualService
If you need to copy a secret from one namespace to another, you will get an error because the ‘namespace’ is still embedded in the yaml and cannot be overridden with the final apply. By using a sed replacement as a filter, you can do a quick transformation and get your desired result. Below is … Kubernetes: copying a secret from one namespace to another
Whether you are running a docker daemon on a development host or a GKE worker node using Docker as the container engine, it is important to understand the amount of disk storage being utilized by the containers. If you navigate into the ‘/var/lib/docker/overlay2’ directory, you will see cryptic hashed ids representing the containers layers instead … Docker: determining container responsible for largest overlay directories
It is typically straightforward to resolve Ansible errors where a simple variable used within an action is not defined. However, when dealing with nested dictionary variables in your Ansible tasks, it can become more complex because not only can the leaf be missing, but also any of the parent container paths. For example, the nested … Ansible: pulling values from nested dictionaries when path might not exist
If you are using vim on a constrained system (container or production system), you may not have the ability to setup vim plugin managers and plugins to mimic a full development environment. However, with a few basic settings in ‘~/.vimrc’, you can get a reasonable rendering that will assist in troubleshooting. ” find list using: … Bash: minimal .vimrc settings for python and yaml editing
Even if the actions in your playbook/role are tagged to separate their logic, this ability to selectively execute will not operate properly when called without any tags because then you will fallback to the special ‘all‘ behavior. Consider a playbook with the following actions. tasks: – debug: msg=”when tag ‘run'” tags: run – debug: msg=”when … Ansible: action only executed if tag set, avoiding ‘all’ behavior
Ansible has support for generating self-signed certificates as well as certificates using a custom root CA (certificate authority). This is possible using the community.crypto collection. I’ve put this into a role named ansible-role-cert-with-ca available on github, and it can be used from a playbook like below: vars: # custom CA, leaving undefined will create self-signed … Ansible: creating SAN certificates with a custom root CA
By default, if an SSH key file is dropped into your personal ‘~/.ssh’ directory that matches a set of standard names, then it will automatically be used as an identity when logging into a remote site (id_rsa, id_dsa, id_ecsda, id_ed25519, or identity). For example, this makes it simple to comply with Github’s requirement to use … Ubuntu: loading a key into ssh-agent at login with a user-level systemd service
An often unexpected behavior of the ‘read‘ utility within a ‘while’ statement is that the last value will not be processed if it does not end with the delimiter. For example, if you are using while to read multiple lines and the last line ends with just EOF (instead of a linefeed followed by EOF), … Bash: while statement with ‘read’ not processing last value
Instead of stringing together sed multiple times in a pipeline, it is also possible to make multiple substitutions with a single invocation of sed. Consider the following example which replaces the word ‘hello’ as well as ‘quick’ in the paragraph: $ sed “s/hello/goodbye/g; s/quick/slow/g” <<EOF hello, world! hello, universe! the quick brown fox EOF goodbye, … Bash: performing multiple substitutions with a single sed invocation
There are scenarios where instead of being explicit about each file name when generating templates from a role, you may want to take all the suffixed files in the ‘templates’ directory and dump their generated content into a destination directory. As a quick example, if you are in a role and want every script in … Ansible: generating content for all template files using with_fileglob
curl has a “write-out” ability to display information after a completed transfer that can be used to support Bash scripting. In it’s simplest form, a curl request looks like this: page=”https://www.google.com” curl $page But this leaves a lot to be desired in a scripting scenario: curl does not exit with a non-zero code on failure, … Bash: Capturing HTTP status code using curl write-out
Creating a ConfigMap using ‘kubectl create configmap’ is a straightforward operation. However, there is not a corresponding ‘kubectl apply’ that can easily update that ConfigMap. As an example, here are the commands for the creation of a simple ConfigMap using a file named “ConfigMap-test1.yaml“. $ cat ConfigMap-test1.yaml test1: foo: bar # create and then show … Kubernetes: Updating an existing ConfigMap using kubectl replace
Although the GCP console provides a manual interface for creating service accounts and assigning roles, it can also be done via the gcloud CLI. Using gcloud, even the json key file for the service account can be generated, which is essential for automation. The full Bash script, create_serviceaccount.sh can be found on github. But here … GCP: Creating gcp service account with IAM roles using gcloud
The Ansible file module will allow you to use ‘state: absent’ to remove a file path, but if you need to perform this action if and only if the path is a symbolic link then you will need to register a test for this condition. By first using the stat module to test if the … Ansible: deleting a file path, but only if a symbolic link
Even though Python2 reached EOL in January 2020, there are still lagging applications and modules that force its continued use on certain systems. If that is your situation, you need the ability to force preference to Python3 unless a program explicitly invokes Python2. If you are using Ubuntu Focal 20, then you have an easy … Python: Setting the preferred Python version on Bionic 18 and Focal 20
Using ldapsearch to query against the insecure port of a Windows Domain Controller is straightforward. However, it can be challenging to get all the pieces in place for a production environment where the secure port must be used and the root CA certificate is typically not from a public CA. Assuming the standard insecure port … Ubuntu: using ldapsearch to query against a secure Windows Domain Controller
If you have multiple values coming through the Bash input pipeline, it can be difficult to process these into a complex, formatted set of arguments unless you use intermediate temporary files. But one way to neatly put together a complex set of arguments from an input pipeline with multiple values is to use awk printf … Bash: using multiple values from an input pipeline to construct and execute a command
As with any programming language, Python has a multitude of ways to accomplish the same task. In this article, I will explore the idea of taking a string and checking if it ‘startswith’ any of the strings from a predetermined list. As context for the example code, RFC1918 sets out several IPv4 ranges that can … Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda
If there is a Kubernetes event that causes mass eviction of pods, it is important to understand the root cause to rule out real issues in the system. However, at some point in your investigation you will want to remove the evicted pods so they do not lead to exhaustion of IP addresses and also … Kubernetes: deleting all evicted pods using kubectl
It is common for a virtualized Guest OS base image to have a generic storage capacity. This capacity can easily be exceeded by production scenarios, performance testing, logging, or even the general cruft of running a machine 24×7. If your VM is using Logical Volume Management (LVM), adding space can be done by following a … Ubuntu: Extending capacity of an LVM volume group using an existing or new disk
It is not uncommon to find an installation page where you are asked to download a packaged release from the github site. But even if you are using the “latest” alias to grab the most recent, you may still need to know the exact release tag for unzipping, configuration management, etc. Luckily, a simple query … Bash: Determining latest github release tag and version
One of the most challenging aspects of using Terraform is dealing with external changes and sprawl of dependent objects that may originate outside your control. Terraform wants to be a system of record and have everything documented in its state as resource/data, however keeping your state in sync when other groups may be doing automation … Terraform: Using non-authoritative resources to avoid IAM membership dependency web
If you are working in a gcp project where the VPC networks are homed within the project itself, then specifying a subnet in gcloud calls is simple, just use the name of the subnet (e.g. ‘default’). However, if you are working in a gcp project where the VPC networks are shared (not owned) with the … GCP: retrieving the full subnet qualification from a shared VPC network
If a git repository requires credentials to clone, and you are still using a username/password (instead of ssh key), it is still possible to have the repository cloned in your automation scripts without be prompted. You just have to ensure that the username and password are properly URL encoded. From the command line, the syntax … Ansible: cloning a git repository that requires credentials
If you need to bootstrap a GCP project’s infrastructure, one of the first things you will want is a service account. The creation of the service account, creating its key, and then assigning binding roles can all be done from the GCP console but for scripting purposes can also be done using the gcloud utility. … GCP: Using gcloud to create and configure a service account
Although the GCP console provides a nice interface for displaying which user/service account is in which IAM security role (IAM & Admin > IAM), it can be difficult to analyze using gcloud get-iam-policy because of the inner array of ‘members’ returned. However, if you use the flattening ability of gcloud, it becomes much easier to … GCP: Analyzing members of IAM role using gcloud filtering and jq
Update July 2021: I have seen errors with external snapshots of volumes on versions of QEMU/KVM/libvirt from Ubuntu 20 Focal. Adding note on using internal snapshot on volume backed by qcow2. Internal snapshots created on QEMU copy-on-write (qcow2) disks are the most commonly used snapshot when using libvirt. It is easy to see why; … KVM: creating and reverting libvirt external snapshots
grep has support for Perl compatible regular expressions (PCRE) by using the -P flag, and this provides a number of useful features. In this article, I’ll show how LookBehind and LookAhead regular expression support can provide enhanced parsing abilities to your shell scripts. For example, consider an xml file “test.xml” with the contents: <root> <path>/my/data</path> … Bash: grep with LookBehind and LookAhead to isolate desired text
A centralized authentication/authorization mechanism is a key part of a security stance as well as a convenience to end users. There are multiple packages and systems to achieve this, and in this article I will focus on integrating back into Windows Active Directory using SSSD for login and group membership. When complete, you will have … Ansible: Login to Ubuntu with Windows Active Directory using SSSD
One of the features of the ‘lineinfile‘ regexp parameter is the ability to use regular expression capture groups in the line output. That allows you to extract values on a found line when constructing the output line. Specifically, that can mean pulling information such as hostname/port, file path, or preserving the yaml indentation of an … Ansible: regex capture groups with lineinfile to preserve yaml indentation
If your Ansible automation includes modifying an existing configuration file (versus managing your own fully templatized version of the config), then you will need to account for variations in that existing file when using ‘lineinfile‘ for key/value pairs. A naive lineinfile replacement will not find commented-out keys and might not even find valid keys that … Ansible: lineinfile with regex to robustly populate key/value pairs in config file