Ansible uses ssh to configure its target host inventory, but for on-premise datacenters as well as hyperscalers like EC2/GCP/Azure, the target hosts are often purposely located in deeper private subnets that cannot be reached from the Ansible orchestrator host. One solution is to enable a bastion/jumpbox host that serves as the forwarding host. It sits … Ansible: orchestrating ssh access through a bastion host
If you can conform to a bit of naming convention, one way to easily render template files in Bash is to have the template and bash variables names match. Then you can either use sed or envsubst to do the substitution. For example, here are two variables being defined and then placeholders embedded into the … Bash: render template from matching bash variables
Managing resources in Azure from the command line can be done natively from Ubuntu using the Azure CLI. First, add the prerequisite packages. sudo apt-get update sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg -y Then install the Microsoft signing key and add the custom repository. curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg –dearmor | sudo tee … Azure: installing the Azure CLI on Ubuntu
You can bake a startup script directly into the creation of your GCE compute instance when using Terraform. Although complex post-configuration should be left to tools such as Ansible, essential bootstrap type commands or custom routes for instances in private subnets are reasons why you might need to use this hook. Below is an example … Terraform: invoking a startup script for a GCE google_compute_instance
You can bake a startup script directly into the creation of your EC2 instance when using Terraform. Although complex post-configuration should be left to tools such as Ansible, essential bootstrap type commands or custom routes for instances in private subnets are reasons why you might need to use this hook. Below is an example of … Terraform: invoking a startup script for an EC2 aws_instance
With Ansible, it is possible to have a variable populated with either the content of a remote or local file. For a file on a remote host, you can use the ‘slurp‘ module to put the content into a register, but then you must base64 decode the content. – name: get content of remote file … Ansible: creating a variable from a remote or local file content
In a single Ansible playbook, you may wish to apply some roles to all hosts, while limiting other roles to only certain groups. While it is certainly possible to apply a role to all hosts and then use a ‘when’ to filter down to the desired group like below: – hosts: all gather_facts: yes roles: … Ansible: applying roles to certain groups in a single playbook
If you have multiple terraform projects, it can be necessary to support multiple versions of the terraform binary to match module and provider dependencies. Instead of creating a custom solution of binary copies and links, this can be done using the Alternatives concept which handles these symbolic links in a standard way using links in … Terraform: using update-alternatives to manage multiple terraform binaries
For Ubuntu, there are a couple of ways you can install the linux-headers package matching the kernel version. You can either explicitly specify the version, or use the meta package as shown below. # specify kernel version using subshell sudo apt-get install -y linux-headers-$(uname -r) # OR meta package that auto-matches kernel sudo apt-get install … Ansible: installing linux-headers matching kernel for Ubuntu
In a previous post, I described the Kubernetes Downward API and how it allows us to inject pod/container metadata into our runtime container. In this article, I’ll show how you can read the environment variables and mounted files from inside a containerized GoLang based application.
In a previous post, I described the Kubernetes Downward API and how it allows us to inject pod/container metadata into our runtime container. In this article, I’ll show how you can read the environment variables and mounted files from inside a containerized Python based application.
The Kubernetes Downward API allows a pod to get access to metadata about itself and the cluster without creating a tight coupling to the Kubernetes API. For example, information such as pod name, labels, annotations, IP address, node, and cpu/memory limits can be made available inside the pod. In this article, I’ll show how to … Kubernetes: using the Downward API to access pod/container metadata
In a previous article I discussed the advantages to keeping container images in the private Google Container Registry of a project. And if you have a GKE cluster in the exact same project, then image pulls happen seamlessly without any additional configuration required. However, if the GKE cluster is in a different project than the … GCP: pulling an image from the Container Registry of another project
Docker hub now enforces pull rate limits (since November 2020). And unfortunately, this limit is often reached at critical moments such as upgrades or infrastructure events when bulk pod recreation is happening. One way to avoid this problem is to place your images into an alternate image registry. This could mean a lot of work … GCP: pushing GKE images into gcr.io to avoid pull rate limits
If you are doing a substitution with sed but need to exclude a specific line or pattern, that can be accomplished by prefixing an exclusion and using “!”. For example, to substitute “hello” for “goodbye” in the following content, but to skip any line containing “world”, use syntax like the following: $ sed “/world/! s/hello/goodbye/g” … Bash: sed substitution with an exclusion pattern
If you need to determine the version of the nginx ingress controller deployed, then you can invoke the ingress controller binary with the ‘–version’ flag. But this binary is located in the ingress-nginx-controller pod, so do a ‘kubectl exec’ like below. # namespace of your nginx ingress ingress_ns=”default” # find running pod podname=$(kubectl get pods … Kubernetes: detecting the installed version of nginx ingress
Pulling the most recent script/file/asset from a remote host may be preferable, but you would like to provide a fallback to a local file just in case the remote pull fails. This is relatively easy to do with Ansible by allowing the remote pull to ‘ignore_errors’ and then doing a local file copy if an … Ansible: preferring a pull from a URL with fallback to a local file
If you are executing gkectl commands that scale or modify your worker nodes and hit a problem, the first place to go is into the gkectl logs. But if you need to dig deeper there are a couple of CRD that can assist in troubleshooting. First, make sure you have executed gkectl with high verbosity … GCP: troubleshooting nodepool replica changes for Anthos on-premise
Once you introduce an istio sidecar proxy into your deployment, it becomes another point at which you might need to troubleshoot network connectivity to the primary container. Assuming you have deployed a pod with an app label “helloworld” in the default namespace listening on port 5000, you can use a command like the following to … Kubernetes: testing pod communication directly from istio sidecar proxy
If your istio ingress Gateway is in a different namespace than your VirtualService, then you need to make sure you prefix the gateway reference with that namespace. For example, if your istio ingress Gateway is in the ‘default’ namespace, yet your Deployment, Service, and VirtualService are in the namespace ‘helloworld’. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: … Kubernetes: istio Gateway in a different namespace than VirtualService
If you need to copy a secret from one namespace to another, you will get an error because the ‘namespace’ is still embedded in the yaml and cannot be overridden with the final apply. By using a sed replacement as a filter, you can do a quick transformation and get your desired result. Below is … Kubernetes: copying a secret from one namespace to another
Whether you are running a docker daemon on a development host or a GKE worker node using Docker as the container engine, it is important to understand the amount of disk storage being utilized by the containers. If you navigate into the ‘/var/lib/docker/overlay2’ directory, you will see cryptic hashed ids representing the containers layers instead … Docker: determining container responsible for largest overlay directories
It is typically straightforward to resolve Ansible errors where a simple variable used within an action is not defined. However, when dealing with nested dictionary variables in your Ansible tasks, it can become more complex because not only can the leaf be missing, but also any of the parent container paths. For example, the nested … Ansible: pulling values from nested dictionaries when path might not exist
If you are using vim on a constrained system (container or production system), you may not have the ability to setup vim plugin managers and plugins to mimic a full development environment. However, with a few basic settings in ‘~/.vimrc’, you can get a reasonable rendering that will assist in troubleshooting. ” find list using: … Bash: minimal .vimrc settings for python and yaml editing
Even if the actions in your playbook/role are tagged to separate their logic, this ability to selectively execute will not operate properly when called without any tags because then you will fallback to the special ‘all‘ behavior. Consider a playbook with the following actions. tasks: – debug: msg=”when tag ‘run'” tags: run – debug: msg=”when … Ansible: action only executed if tag set, avoiding ‘all’ behavior
Ansible has support for generating self-signed certificates as well as certificates using a custom root CA (certificate authority). This is possible using the community.crypto collection. I’ve put this into a role named ansible-role-cert-with-ca available on github, and it can be used from a playbook like below: vars: # custom CA, leaving undefined will create self-signed … Ansible: creating SAN certificates with a custom root CA
By default, if an SSH key file is dropped into your personal ‘~/.ssh’ directory that matches a set of standard names, then it will automatically be used as an identity when logging into a remote site (id_rsa, id_dsa, id_ecsda, id_ed25519, or identity). For example, this makes it simple to comply with Github’s requirement to use … Ubuntu: loading a key into ssh-agent at login with a user-level systemd service
An often unexpected behavior of the ‘read‘ utility within a ‘while’ statement is that the last value will not be processed if it does not end with the delimiter. For example, if you are using while to read multiple lines and the last line ends with just EOF (instead of a linefeed followed by EOF), … Bash: while statement with ‘read’ not processing last value
Instead of stringing together sed multiple times in a pipeline, it is also possible to make multiple substitutions with a single invocation of sed. Consider the following example which replaces the word ‘hello’ as well as ‘quick’ in the paragraph: $ sed “s/hello/goodbye/g; s/quick/slow/g” <<EOF hello, world! hello, universe! the quick brown fox EOF goodbye, … Bash: performing multiple substitutions with a single sed invocation
There are scenarios where instead of being explicit about each file name when generating templates from a role, you may want to take all the suffixed files in the ‘templates’ directory and dump their generated content into a destination directory. As a quick example, if you are in a role and want every script in … Ansible: generating content for all template files using with_fileglob
curl has a “write-out” ability to display information after a completed transfer that can be used to support Bash scripting. In it’s simplest form, a curl request looks like this: page=”https://www.google.com” curl $page But this leaves a lot to be desired in a scripting scenario: curl does not exit with a non-zero code on failure, … Bash: Capturing HTTP status code using curl write-out
Creating a ConfigMap using ‘kubectl create configmap’ is a straightforward operation. However, there is not a corresponding ‘kubectl apply’ that can easily update that ConfigMap. As an example, here are the commands for the creation of a simple ConfigMap using a file named “ConfigMap-test1.yaml“. $ cat ConfigMap-test1.yaml test1: foo: bar # create and then show … Kubernetes: Updating an existing ConfigMap using kubectl replace
Although the GCP console provides a manual interface for creating service accounts and assigning roles, it can also be done via the gcloud CLI. Using gcloud, even the json key file for the service account can be generated, which is essential for automation. The full Bash script, create_serviceaccount.sh can be found on github. But here … GCP: Creating gcp service account with IAM roles using gcloud
The Ansible file module will allow you to use ‘state: absent’ to remove a file path, but if you need to perform this action if and only if the path is a symbolic link then you will need to register a test for this condition. By first using the stat module to test if the … Ansible: deleting a file path, but only if a symbolic link
Even though Python2 reached EOL in January 2020, there are still lagging applications and modules that force its continued use on certain systems. If that is your situation, you need the ability to force preference to Python3 unless a program explicitly invokes Python2. If you are using Ubuntu Focal 20, then you have an easy … Python: Setting the preferred Python version on Bionic 18 and Focal 20
Using ldapsearch to query against the insecure port of a Windows Domain Controller is straightforward. However, it can be challenging to get all the pieces in place for a production environment where the secure port must be used and the root CA certificate is typically not from a public CA. Assuming the standard insecure port … Ubuntu: using ldapsearch to query against a secure Windows Domain Controller
If you have multiple values coming through the Bash input pipeline, it can be difficult to process these into a complex, formatted set of arguments unless you use intermediate temporary files. But one way to neatly put together a complex set of arguments from an input pipeline with multiple values is to use awk printf … Bash: using multiple values from an input pipeline to construct and execute a command
As with any programming language, Python has a multitude of ways to accomplish the same task. In this article, I will explore the idea of taking a string and checking if it ‘startswith’ any of the strings from a predetermined list. As context for the example code, RFC1918 sets out several IPv4 ranges that can … Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda
If there is a Kubernetes event that causes mass eviction of pods, it is important to understand the root cause to rule out real issues in the system. However, at some point in your investigation you will want to remove the evicted pods so they do not lead to exhaustion of IP addresses and also … Kubernetes: deleting all evicted pods using kubectl
It is common for a virtualized Guest OS base image to have a generic storage capacity. This capacity can easily be exceeded by production scenarios, performance testing, logging, or even the general cruft of running a machine 24×7. If your VM is using Logical Volume Management (LVM), adding space can be done by following a … Ubuntu: Extending capacity of an LVM volume group using an existing or new disk