yq: replace section of one yaml file with content section of another


Categories: Linux, Scripting

Mike Farah’s yq yaml processor has a a rich set of operators and functions for advanced usage.  In this article, I will illustrate how to take a block of yaml from one file to populate a specific location in another.

Install yq

Here are the steps for installing the latest yq on Snap enabled systems like Ubuntu/Debian.  See the notes at bottom if you need manual installation instead.

sudo snap install yq
yq --version

Sample yaml

We will use the following yaml files to illustrate copying the section of one yaml file to another.

cat <<EOF >company-servers.yaml
company:
  regions: [ 'us-east', 'us-central', 'us-west']
  inventory:
    machineA:
      region: us-east
      tags:
      - linux
    machineB:
      region: us-central
      tags:
      - windows
EOF

cat <<EOF >company-regions-new.yaml
regions-new: ['us-east', 'us-central', 'us-west', 'eu-1', 'eu-2']
EOF

Copying a section from one file to another

If you wanted the ‘.regions-new’ element from ‘company-regions-new.yaml’ to replace the ‘regions’ section in ‘company-servers.yaml’, then you could use eval-all and the fileIndex like below.

$ yq eval-all 'select(fileIndex==0).company.regions = select(fileIndex==1).regions-new | select(fileIndex==0)' company-servers.yaml company-regions-new.yaml

company:
  regions: ['us-east', 'us-central', 'us-west', 'eu-1', 'eu-2']
  inventory:
    machineA:
      region: us-east
      tags:
        - linux
    machineB:
      region: us-central
      tags:
        - windows

Notice the ‘regions’ now include the European ‘eu-1’ and ‘eu-2’ regions that did not exist before.

Github project

The examples shown above can be pulled down from my github project.

myproject=https://raw.githubusercontent.com/fabianlee/blogcode
wget $myproject/master/yq/update-filesection/company-servers.yaml
wget $myproject/master/yq/update-filesection/company-regions-new.yaml
wget $myproject/master/yq/update-filesection/yq-mikefarah-merge-two-files.sh
chmod +x *.sh

# run script
./yq-mikefarah-merge-two-files.sh

 

REFERENCES

yq github project

yq manual pages

yq show and tell, examples

NOTES

Manual download and installation

# essential packages
sudo apt get install curl jq wget -y

# get latest release version
latest_yq_linux=$(curl -sL https://api.github.com/repos/mikefarah/yq/releases/latest | jq -r ".assets[].browser_download_url" | grep linux_amd64.tar.gz)

# download, extract, and put binary into PATH
wget $latest_yq_linux
tar xvfz yq_linux_amd64.tar.gz
sudo cp yq_linux_amd64 /usr/local/bin/yq