If you are using awk with a relatively indexed NF variable (number of fields), and get an error like below, this is because the input being parsed does not have the number of fields expected. awk: run time error: negative field index $-1 The root problem is the awk expression was expecting a certain number … Bash: resolving awk run time error, negative field index
The openssl utility can be used to show the details of a certificate, including its ‘Not After’ expiration date in string format. This can be transformed into “how many days till expiration” with a bit of Bash date math. Create test certificate and key Using a line provided by Diego Woitasen for non-interactive self-signed certification … Bash: calculating number of days till certificate expiration using openssl
The first action you typically take after “git clone” is to change into the newly created directory. This can be accomplished at the Bash shell in a couple of ways. Here is the git URL we will use as an example for this article. # can be https or ssh, can end in .git or … Bash: change into directory just created with git clone
A simple grep can help you recursively search for a substring in all the files of a directory. Here is an example of looking for multi-doc yaml files: # recursively search directory, looking for line ‘—‘ # regex: caret means line starts with, dollar sign mean line ends with grep -sr ‘^—$’ If you have … Bash: counting number of times substring is found in directory
Listing all the pods belonging to a deployment can be done by querying its selectors, but using the deployment’s synthesized replicaset identifier allows for easier automation. # deployment name and namespace deployment_name=mydeployment deployment_ns=mynamespace # get replica set identifier for deployment dep_rs=$(kubectl describe deployment $deployment_name -n $deployment_ns | grep ^NewReplicaSet | awk ‘{print $2}’) # get … Kubernetes: list all pods in deployment
If you need to extract the Nth occurrence of a match in a file (given definitive block separators), awk provides a convenient way to express the extraction. For example, a chained pem certificate will have multiple certification definitions with unique starting and ending marker lines. Here is how you would extract the second certificate. awk … Bash: awk to extract Nth match from file based on line separator
Mike Farah’s yq yaml processor has a a rich set of operators and functions for advanced usage. In this article, I will illustrate how to update a deeply nested element in yaml. Install yq Here are the steps for installing the latest yq on Snap enabled systems like Ubuntu/Debian. See the notes at bottom if … yq: update deeply nested elements in yaml
Mike Farah’s yq yaml processor has a a rich set of operators and functions for advanced usage. In this article, I will illustrate how to take a block of yaml from one file to populate a specific location in another. Install yq Here are the steps for installing the latest yq on Snap enabled systems … yq: replace section of one yaml file with content section of another
Gitlab has an official CLI tool that allows you to perform many of the operations you may currently perform in the web UI. Among its many uses, the ‘glab‘ utility will allow you to fork repositories, manage issues, merge PR, and create releases all from the console for ease of use or automation. Below I … GitLab: glab official CLI tool for repository operations
The GitHub “Release” page for a repository can provide your consumers a convenient way to download a binary version of your software as well as track the latest changes and enhancements. In this article, I will show how to invoke a local release process for a Go language binary. A new Github release will be … Github: locally invoked release process for a Go binary
If you need to quickly standup a secure HTTPS server on a machine, perhaps to validate that a client is sending the proper method or parameters, the socat utility can assist. In this article we will use “socat.local” as the FQDN of our certificate and host. Maybe you already have a pem key+certificate for this … Linux: socat used as secure HTTPS web server
In environments where certificates are manually deployed, reloading TLS certs is often only done annually when the certificate is near expiration. This long lapse in time often means that someone else has inherited the task of renewal, and the original key in use may even be in question. Luckily, openssl provides a way to validate … Linux: openssl to validate whether private key and TLS certificate match
sed is an excellent utility for doing substitutions, but if you need to work across multiple files, then you need to pair it with a command that can provide a list of candidate files. This is where find comes in handy. For example, to replace all instances of ‘zebra’ with ‘horse’ recursively in the current … Linux: sed to replace across multiple files in directory
When using a private key on the client to ssh into a remote server with the matching public certificate in ~/.ssh/authorized_keys, a common failure message from the client is: Permission denied (publickey) The most common reasons for this is private key permissions issues (chmod 600), a misconfiguration of authorized_keys, or trying to send the wrong … Linux: ssh-keygen to check whether ssh private key and public cert are keypair
GCP Compute VM Instances can be set to run as a service account with API scopes that allow specific operations to be performed. If you need to change the service account or the scopes, you will need to power down the VM instance, make the changes, and then start the VM instance up again. Here … GCP: gcloud to change VM instance service account and API scope
If you are scripting a set of gcloud commands, there is a good chance that it is within a Bash script. While some Bash loops can be driven by a single variable, sometimes you need access to multiple variables within the loop. In this article I will show you can drive a Bash loop requiring … GCP: gcloud csv format with no-heading for Bash parsing
When GCP operations fail due to permissions issues, checking the IAM roles assigned to a user, group, or service account becomes a necessity. When hierarchical projects and organizations are involved it becomes even more complex. This article will show you how to use gcloud at the project and organization level to pull IAM policies for … GCP: listing IAM roles for user, group, and service account in project and organization
Windows AD FS provides enterprise Identity and Authentication services, which includes support for OAuth2 and OIDC authentication flows. In this article, we will create and configure an ADFS Application group that supports the Authorization Code flow. This flow allows an application to access a 3rd party API on behalf of the end user as illustrated … Microsoft: configuring an Application Group for OAuth2/OIDC on ADFS 2019
If you have a Kubernetes yaml manifest that contains multiple documents, targeting a single document for modification while still outputting the other documents untouched can be a challenge. As an example, consider the simple example below were you have a single yaml file that contains: a Namespace, Deployment, and DaemonSet. And we want to add … Kubernetes: targeting the addition of array items to a multi-document yaml manifest
Bash scripts will often assume that they are being invoked from the same directory where they are located. The script may use relative paths to configuration files or logs that it expects. The problem is that one cannot assume that the directory a script lives in is necessarily the one it is being invoked from. … Bash: current directory versus directory of script
If you have a Bash script that needs to be sourced (versus directly executed), you can enforce this by checking the execution context in the script. Here is an example of setting a flag that can be evaluated and then acted upon. (return 0 2>/dev/null) && sourced=1 || sourced=0 if [ $sourced -eq 0 ]; … Bash: test whether script is invoked directly or sourced
GitHub has a CLI tool that allows you to perform many of the operations you may currently perform in the web UI. Among its many uses, the ‘gh‘ utility will allow you to fork repositories, modify issues, merge PR, and create releases all from the console for ease of use or automation. Below I will … GitHub: CLI tool for repository operations
With kustomize built into the kubectl CLI since version 1.14, there is little reason not to take advantage of this overlay system to deploy components to your Kubernetes cluster. Kustomize has the advantage that it is purpose built to understand and validate yaml and Kubernetes CRD, as opposed to bespoke templating solutions using sed/envsubst, Ansible, … Kubernetes: kustomize overlay to enrich a base resource
The Compute Engine default service account is automatically generated for your project with the Editor role, and by default is attached to all VM instances created in the project. You can pull the exact id using gcloud. gcloud iam service-accounts list –filter=”displayName:’Compute Engine default service account'” –format=’value(email)’ The syntax will be ${project_id}-compute@developer.gserviceaccount.com. If you want … GCP: VM instances running as the Compute Engine default service account
A Bash script can often run into the situation where a utility in the pipeline creates a file, but because of an unexpected error, the size of the resulting file is zero bytes. And if that situation is not prepared for, the mere existence of the file might signal success to the following commands in … Bash: test both file existence and size to avoid signalling success
For bulk administration of hosts via ssh, there are powerful tools such as Ansible for the job. However, if you need to automate ssh for a one-off task, you may find the remote host systems throwing up interactive authentication prompts for sudo that need to be answered. If you need to run sudo commands that … Bash: automating ssh login and sudo that require interactive login
Killing a defunct zombie process means identifying and killing the parent process. In this article, I will show how you can force a zombie process so that you can practice removing it. A simple way of forcing a zombie process is to start a process that has a short sleep in the background, immediately invoking … Bash: identifying and killing a zombie child processes
Handling signals from within Bash has a simple syntax and can be used to handle user interruption with Ctrl-C (SIGINT), as a cleanup hook at the end of processing (SIGEXIT), or even as a custom messaging mechanism (SIGUSR1). Trapping a signal is as easy as specifying a custom function name and the signal name as … Bash: trapping signals during script processing
When calling processes within a script, there may be situations where you need to put a time limit on the call due to significant blocking times. Perhaps a DNS or network call would block if certain firewall rules do not exist yet, or a long-running API call would need to be considered in error if … Bash: using timeout to put time limit on invoked commands
If the logic in your Bash script needs to force the last exit code variable “$?”, you can accomplish that with an exit inside a subprocess. For example, here is a script snippet. $(exit 99) echo “last process exit code was $?” This would return the output below. last process exit code was 99 This … Bash: forcing the process exit code using a subshell
GitHub has a feature that allows other members to follow your account and receive alerts on actions you take in any repository. Unfortunately, there is not a setting that allows you to disable this at a global level. The only workaround is to temporarily block the user, which removes the relationship. This can be done … GitHub: removing followers with the GitHub api
Most of the modern cloud platforms and utilities have us manipulate either JSON or YAML configuration files. And when you start dealing with real world scenarios with hundreds of lines of embedded data structures it is too difficult and error-prone to manually inspect indentation levels to determine the exact dotted or json path to an … Python: converting JSON to dot notation for easier path determination
If you are using a Terraform “for_each” and get the error message below, it is most likely because you are sending an ordered list instead of an unordered set (which is not supported at the resource level). The given “for_each” argument value is unsuitable: the “for_each” argument must be a map, or set of strings, … Terraform: converting ordered lists to sets to avoid errors with for_each
If you have captured the multi-line output of a previous Bash command in a variable and later need to count the number of lines stored in that variable, you may reach for an apparently easy solution first. But there are corner cases you may not have covered. The first syntax gotcha is to make sure … Bash: count number of lines in previously captured stdout
Terraform is a popular tool for provisioning infrastructure on cloud providers such as EC2, Azure, and GCP. If you want to install Teraform on Ubuntu using apt-get, follow HashiCorp’s standard installation document. However, I find that I often need multiple versions for different projects. Find your desired version of the binaries at the Terraform download … Terraform: installing Terraform manually on Ubuntu
If you can conform to a bit of naming convention, one way to easily render template files in Bash is to have the template and bash variables names match. Then you can either use sed or envsubst to do the substitution. For example, here are two variables being defined and then placeholders embedded into the … Bash: render template from matching bash variables
If you have multiple terraform projects, it can be necessary to support multiple versions of the terraform binary to match module and provider dependencies. Instead of creating a custom solution of binary copies and links, this can be done using the Alternatives concept which handles these symbolic links in a standard way using links in … Terraform: using update-alternatives to manage multiple terraform binaries
If you are doing a substitution with sed but need to exclude a specific line or pattern, that can be accomplished by prefixing an exclusion and using “!”. For example, to substitute “hello” for “goodbye” in the following content, but to skip any line containing “world”, use syntax like the following: $ sed “/world/! s/hello/goodbye/g” … Bash: sed substitution with an exclusion pattern
An often unexpected behavior of the ‘read‘ utility within a ‘while’ statement is that the last value will not be processed if it does not end with the delimiter. For example, if you are using while to read multiple lines and the last line ends with just EOF (instead of a linefeed followed by EOF), … Bash: while statement with ‘read’ not processing last value
Instead of stringing together sed multiple times in a pipeline, it is also possible to make multiple substitutions with a single invocation of sed. Consider the following example which replaces the word ‘hello’ as well as ‘quick’ in the paragraph: $ sed “s/hello/goodbye/g; s/quick/slow/g” <<EOF hello, world! hello, universe! the quick brown fox EOF goodbye, … Bash: performing multiple substitutions with a single sed invocation