sed is an excellent utility for doing substitutions, but if you need to work across multiple files, then you need to pair it with a command that can provide a list of candidate files. This is where find comes in handy. For example, to replace all instances of ‘zebra’ with ‘horse’ recursively in the current … Linux: sed to replace across multiple files in directory
If you are scripting a set of gcloud commands, there is a good chance that it is within a Bash script. While some Bash loops can be driven by a single variable, sometimes you need access to multiple variables within the loop. In this article I will show you can drive a Bash loop requiring … GCP: gcloud csv format with no-heading for Bash parsing
The ClientAliveInterval and ClientAliveMaxCount settings in “/etc/sshd/sshd_config” work together to control the timeout value of an ssh session on the server side. But under BASH, to keep idle client sessions from timing out, you also need to set the ‘TMOUT’ variable or you will see messages like below when disconnected. timed out waiting for input: … Bash: extend timeout for idle ssh sessions using TMOUT
Bash scripts will often assume that they are being invoked from the same directory where they are located. The script may use relative paths to configuration files or logs that it expects. The problem is that one cannot assume that the directory a script lives in is necessarily the one it is being invoked from. … Bash: current directory versus directory of script
If you have a Bash script that needs to be sourced (versus directly executed), you can enforce this by checking the execution context in the script. Here is an example of setting a flag that can be evaluated and then acted upon. (return 0 2>/dev/null) && sourced=1 || sourced=0 if [ $sourced -eq 0 ]; … Bash: test whether script is invoked directly or sourced
A Bash script can often run into the situation where a utility in the pipeline creates a file, but because of an unexpected error, the size of the resulting file is zero bytes. And if that situation is not prepared for, the mere existence of the file might signal success to the following commands in … Bash: test both file existence and size to avoid signalling success
Handling signals from within Bash has a simple syntax and can be used to handle user interruption with Ctrl-C (SIGINT), as a cleanup hook at the end of processing (SIGEXIT), or even as a custom messaging mechanism (SIGUSR1). Trapping a signal is as easy as specifying a custom function name and the signal name as … Bash: trapping signals during script processing
When calling processes within a script, there may be situations where you need to put a time limit on the call due to significant blocking times. Perhaps a DNS or network call would block if certain firewall rules do not exist yet, or a long-running API call would need to be considered in error if … Bash: using timeout to put time limit on invoked commands
If the logic in your Bash script needs to force the last exit code variable “$?”, you can accomplish that with an exit inside a subprocess. For example, here is a script snippet. $(exit 99) echo “last process exit code was $?” This would return the output below. last process exit code was 99 This … Bash: forcing the process exit code using a subshell
If you have a Bash script that needs to run periodically, you can run it using a crontab entry. But you can also have it invoked by Systemd using systemd.timer. Furthermore, you can run Systemd services as user-level services instead of the typical system-level service for even further isolation. Running via Systemd provides more powerful … Ubuntu: Running a bash script periodically with a user-level Systemd timer
If you have a Bash script that needs to run periodically, you can run it using a crontab entry or file. But you can also have it invoked from Systemd using systemd.timer. Running via Systemd provides more powerful constructs for invocation, configuration, monitoring, and logging. In this article, I will show how to periodically run … Ubuntu: Running a bash script periodically with a system-level Systemd timer
If you have a file with special characters (single quotes, wildcard, etc) in the name, it can be difficult to discover the exact escape sequence to correctly delete. To avoid playing with escape characters, you can simply use the inode number of the file instead. For example, let’s say you accidentally specify tar options incorrectly … Bash: deleting a file with special characters using its inode value
If you are creating a VM resource and must run a Bash script as part of the initialization, that can be done within Terraform using the remote-exec provisioner and its ability to execute scripts via ssh. If you need to send arguments to this script, there is a standard pattern described in the official documentation … Terraform: post-configuration by calling remote-exec script with parameters
If you have captured the multi-line output of a previous Bash command in a variable and later need to count the number of lines stored in that variable, you may reach for an apparently easy solution first. But there are corner cases you may not have covered. The first syntax gotcha is to make sure … Bash: count number of lines in previously captured stdout
Needing to find the most recently modified files in a directory is a pretty common need. Luckily the find utility has flags to easily explore a directory recursively and list recently modified files. If you want to find modified files within ‘N’ days ago from the current directory. # files within the last 24 hours … Bash: find most recently modified files
If you need to create a file that has the exact same ownership and permission bits as an existing file, the ‘reference’ flag provides a convenient shortcut. For example, if you had a file named ‘myoriginal’ that had the exact ownership and permissions required for a new file ‘mynewfile’, you could use the commands below … Bash: cloning the ownership and permissions of another file using reference
One way to implement character padding in Bash is to use printf and substring extraction. This can be especially useful in reports or menu display. Given a $padding variable that contains the maximum length of characters, you can subtract out the length of a display string like below. # length of maximum padding padding=”………………………………..” printf … Bash: using printf to display fixed-width padded string
If you can conform to a bit of naming convention, one way to easily render template files in Bash is to have the template and bash variables names match. Then you can either use sed or envsubst to do the substitution. For example, here are two variables being defined and then placeholders embedded into the … Bash: render template from matching bash variables
curl has a “write-out” ability to display information after a completed transfer that can be used to support Bash scripting. In it’s simplest form, a curl request looks like this: page=”https://www.google.com” curl $page But this leaves a lot to be desired in a scripting scenario: curl does not exit with a non-zero code on failure, … Bash: Capturing HTTP status code using curl write-out
If you have multiple values coming through the Bash input pipeline, it can be difficult to process these into a complex, formatted set of arguments unless you use intermediate temporary files. But one way to neatly put together a complex set of arguments from an input pipeline with multiple values is to use awk printf … Bash: using multiple values from an input pipeline to construct and execute a command
Finding the most recently modified files in a directory can be extremely beneficial when you have been making changes in files throughout the directory structure as part of a work effort, and now need to go back and pinpoint everything that was changed. This command will provide you the 10 most recently modified files, excluding … Bash: deep listing the most recently modified files in a directory
Bash has a rich set of control structures but it is still common to see logic expressions used a shorthand for if-then-else/tertiary statements. This can provide a curt and single-line ingestable way of controlling the flow of your Bash script. For example: $ [ 1 -eq 1 ] && echo “correct, 1 does indeed equal … Bash: Using logic expressions as a shorthand for if-then-else control
If you need to output only lines before (or after) a unique line identified by a regular expression, then here are examples of using sed. All examples in this article can be found in github. Consider the following lines of content in the variable “$lines” $ echo “$lines” ape swings frog jumps logs dog barks … Bash: output all lines before/after line identified by regex
Whether looking at differences in filenames, installed packages, etc. it can be useful to calculate the difference between two Bash arrays. SiegeX on stackoverflow.com offered the following function using awk, and I have built a full example available on github. function arraydiff() { awk ‘BEGIN{RS=ORS=” “} {NR==FNR?a[$0]++:a[$0]–} END{for(k in a)if(a[k])print k}’ <(echo -n “${!1}”) <(echo … Bash: Difference between two arrays
Just as in other programming languages, associative arrays in Bash are useful for search, set management, and keying into a list of values. The label may be different, but whether called “map”, “dictionary”, or “associative array”, the same concepts apply. In its simplest form, here is an initialization and then loop through the key/value pairs. … Bash: Associative array initialization and usage
sed is a powerful utility for transforming text. One of the nice tricks with sed is the ability to reuse capture groups from the source string in the replacement value you are constructing. For example, if you have have the following kernel parameters in “/etc/default/grub” $ grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash” And wanted to append … Bash: Appending to existing values using sed capture group
The =~ binary operator provides the ability to compare a string to a POSIX extended regular expression in the shell. Take note, the right-hand side regex cannot be surrounded by quotes or it will be treated as a regular string, it cannot contain spaces, and must conform to POSIX regex rules and use character classes … Bash: Using BASH_REMATCH to pull capture groups from a regex
Shell parameter expansion provides various ways to manipulate strings, and a convenient way to succinctly express renaming a set of files. In its simplest form, parameter expansion is simply ${parameter}. But look at these examples: $ mystr=”TheQuickBrownFox.jpg” # chop off last 4 digits $ echo ${mystr:0:-4} TheQuickBrownFox # truncate end of string ‘.jpg’ $ echo … Bash: Renaming files using shell parameter expansion
If you are in the middle of a text processing pipeline, and need to insert a shell or environment variable into the output of awk, you can use the “-v” flag. Here are two files containing animal classifications: $ echo -e “shark=fish\ndolphin=mammal” > ocean.txt $ echo -e “dog=mammal\neagle=bird” > land.txt By passing the loop variable … Bash: Using shell or environment variables in awk output
Using head to get the first lines of a stream, and tail to get the last lines in a stream is intuitive. But if you need to skip the first few lines of a stream, then you use tail “-n +k” syntax. And to skip the last lines of a stream head “-n -k” syntax. … Bash: Skipping lines at the top or bottom of a stream
If you need to call attention to a message within your script, a splash of color is often the best way to get noticed in the sea of monochrome text. The standard ANSI escape codes are supported at most consoles, and in its simplest form can be used like this: RED=’\033[0;31m’ GREEN=’\033[0;32m’ NC=’\033[0m’ echo -e … Bash: Outputting text in color for readability
This may come as a surprise, but Bash only supports integer arithmetic natively. If you need to perform calculations on floating point numbers, you will need to call out to a utility program like bc or Python. As a quick proof to yourself, try multiplying integers, then do the same with floating point numbers. $ … Bash: Performing floating arithmetic using bc
A common task when configuring software is modifying a properties file to set or override behavior. And although properties files are simple key=value pairs, there are some corner cases that make it challenging. For example, when setting a key/value pair, the key may be commented out with a hash mark in front of the line. … Bash: setting and replacing values in a properties file use sed
The typical usage for xargs is feeding it multiple lines of input, and having it run commands using each line as an argument. But there are other cases when you have a single line separated by a delimiter or even a list of optionally quoted args, and I’ll show in this article how to handle … Bash: Running command on quoted list of parameters using xargs
The AWS CLI provides a comprehensive command set, but if you want advanced functionality such as checking if an object already exists before you create it, or polling an instance until it is a certain state then you will need a slightly higher level of abstraction. My aws-common-funcs.sh script provides a set of Bash functions … AWS: Bash helper functions for common AWS CLI calls
If you are using the direct output of a command to feed a while loop in BASH, you may still want to take user input inside that loop. For example, if you need the user to confirm each line by pressing <ENTER>. Take a simple example like this: grep one test.txt | while read -r … Bash: Reading input from the console while looping over output of command
If you have Bash console commands that you type out frequently, consider creating an alias. By simply adding a “.bash_aliases” files in your home directory, you can easily define shortcuts to frequent commands. For example, if you frequently list all the files in a directory reverse sorted by date, you may end up typing “ls … Ubuntu: Creating an alias for common commands under bash