GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster


Categories: Automation, DevOps, Hyperscaler

As much as Terraform pushes to be the absolute system of record for resources it creates, often valid external processes are assisting in managing those same resources. Here are some examples of legitimate external changes: Other company-approved Terraform scripts applying labeling to resources in order to track ownership and costs Security teams modifying IAM roles … GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster

GCP: Cloud Run with build trigger coming from remote GitHub repository


Categories: Containers, Development, Hyperscaler

GCP build triggers can easily handle Continuous Deployment (CD) when the source code is homed in a Google Cloud Source repository.  But even if the system of record for your source is a remote GitHub repository, these same type of push and tag events can be consumed if you configure a connection and repository link. … GCP: Cloud Run with build trigger coming from remote GitHub repository

GCP: Cloud Run/Function to handle requests to GKE cluster during maintenance


Categories: Hyperscaler

At some point, there will be a system change significant enough that a maintenance window needs to be scheduled with customers.   But that doesn’t mean the end-user traffic or client integrations will stop requesting the services. What we need to present to end-users is a maintenance page during this outage to indicate the overall solution … GCP: Cloud Run/Function to handle requests to GKE cluster during maintenance

Ansible: adding custom apt repository with ‘signed-by’ gpg key


Categories: Automation

The centralized system keyring for apt was deprecated starting in Ubuntu 21, and is being replaced with an explicit path to the local gpg key in the ‘signed-by’ attribute. I have written more extensive articles on this subject [here,here], but from an Ansible perspective, this means ensuring the gpg key is downloaded to ‘/usr/share/keyrings’ with … Ansible: adding custom apt repository with ‘signed-by’ gpg key

Ansible: generating templates with deep directory structure using with_filetree


Categories: Automation

If you have a simple directory containing multiple template files that should be generated on a target host, the ‘with_fileglob‘ lookup plugin provides an easy way to render them.  Below is an example rendering all the files from the ‘templates’ directory of a role. – name: create file out of every file in template directory … Ansible: generating templates with deep directory structure using with_filetree

GKE: show pod distribution across nodes and zones


Categories: Hyperscaler, Kubernetes

Whether you are working on scaling, performance, or high-availability, it can be useful to see exactly which Kubernetes worker node that pods are being scheduled unto. Pods as distributed across worker nodes ns=default kubectl get pods -n $ns -o=custom-columns=NAME:.metadata.name,NODE:.spec.nodeName Pods as distributed across zones (GKE specific) If you wanted to take it one step further … GKE: show pod distribution across nodes and zones

GKE: upgrade Anthos Config Management for GKE cluster


Categories: Hyperscaler

If you are managing GKE clusters using Anthos Config Management (ACM) and need to take advantage of newer features or enhancements in ConfigSync or PolicyController, upgrading these components can be done using the gcloud utility. # check current version of ACM on GKE clusters gcloud beta container fleet config-management version # select membership to upgrade … GKE: upgrade Anthos Config Management for GKE cluster

Python: fixing ‘CryptographyDeprecationWarning: Blowfish has been deprecated’


Categories: Python

If you are getting a warning similar to below when running a Python3 application: /usr/lib/python3/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated This can be resolved by upgrading to the latest paramiko module. # check current version then upgrade pip3 show paramiko pip3 install paramiko –upgrade # check upgraded version pip3 show paramiko In my case, this … Python: fixing ‘CryptographyDeprecationWarning: Blowfish has been deprecated’

Terraform: migrate state from local to remote Google Cloud Storage bucket and back


Categories: DevOps, Hyperscaler

In this article I will demonstrate how to take a Terraform configuration that is using a local state file and migrate its persistent state to a remote Google Cloud Storage bucket (GCS).  We will then perform the migration again, but this time to bring the remote state back to a local file. We will illustrate … Terraform: migrate state from local to remote Google Cloud Storage bucket and back

GKE: Determine Anthos on-prem GKE master node and IP address


Categories: Hyperscaler, Kubernetes

If you are using Anthos GKE on-premise and need to determine which node of your Admin Cluster is the master, query for the master role.  The label is ‘node-role.kubernetes.io/master’. $ kubectl get nodes -l node-role.kubernetes.io/master NAME STATUS ROLES AGE VERSION gke-admin-master-adfwa Ready control-plane,master 7d v1.24.9-gke.100 # using wide will also show External and Internal IP … GKE: Determine Anthos on-prem GKE master node and IP address

Kubernetes: list all pods in deployment


Categories: Kubernetes, Scripting

Listing all the pods belonging to a deployment can be done by querying its selectors, but using the deployment’s synthesized replicaset identifier allows for easier automation. # deployment name and namespace deployment_name=mydeployment deployment_ns=mynamespace # get replica set identifier for deployment dep_rs=$(kubectl describe deployment $deployment_name -n $deployment_ns | grep ^NewReplicaSet | awk ‘{print $2}’) # get … Kubernetes: list all pods in deployment

OpenWrt: installing dig from opkg


Categories: OpenWrt

For troubleshooting DNS issues, running the dig utility directly from OpenWrt can be essential.  This is easily done by installing the ‘bind-dig’ package as shown below. opkg update opkg install bind-dig

Ubuntu: ‘Connection to the Snap Store failed’ during upgrade from Ubuntu 20 to 22


Categories: Linux

If you are upgrading from Ubuntu 20 to Ubuntu 22 using ‘do-release-upgrade’ and get a fatal error ‘Connection to the the Snap Store failed’, this may be resolved by removing the ‘lxd’ package which is a lightweight container supervisor. sudo /etc/init.d/lxd stop sudo rm -fr /var/lib/lxd sudo dpkg –force depends -P lxd; sudo dpkg –force … Ubuntu: ‘Connection to the Snap Store failed’ during upgrade from Ubuntu 20 to 22

GCP: Google Cloud Storage bucket with permissions for user or service account


Categories: Hyperscaler

Creating a Google Cloud Storage bucket is simple, but the IAM permissions required to perform operations in the bucket can be difficult to understand.  Especially when you want something as simple as to provide upload/download access to the person who created the bucket and perhaps a service account. Below are the commands for creating a … GCP: Google Cloud Storage bucket with permissions for user or service account

Linux: using nmap to check the secure protocols and ciphers of a site


Categories: Linux

While enabling HTTPS is a important step in securing your web application, it is critical that you take steps to disable legacy protocols and low strength ciphers that can circumvent the very security you are attempting to implement. The Qualys SSL test is popular for grading the overall security of a public site, but you … Linux: using nmap to check the secure protocols and ciphers of a site

OpenWrt: bridge VLAN filtering for OpenWrt 21.x with DSA, isolated guest Wi-Fi


Categories: OpenWrt

There were significant changes made to VLAN configuration between OpenWrt 19.x and 21.x.  Also, many of the target chipset were migrated from swconfig to DSA (Distributed Switch Architecture), which introduced differences in bridging. In this article, I will create a set of VLAN for my OpenWrt 21.x DSA-enabled router with isolated guest Wi-Fi networks. I … OpenWrt: bridge VLAN filtering for OpenWrt 21.x with DSA, isolated guest Wi-Fi

Kubernetes: restart a simple pod


Categories: Kubernetes

A pod belonging to a deployment can be manually deleted, scaled down, or restarted to get a fresh pod.  However, if all you have is a simple pod definition, these actions are not available. One way of restarting the pod is to output its full yaml definition and use ‘kubectl replace’ with the force option. … Kubernetes: restart a simple pod

Kubernetes: patch every array element using kubectl and jq


Categories: Kubernetes

Below is an example using ‘kubectl patch’ to update the securityContext of a single, specific container named ‘my-init-container1’ of the ‘initContainers’ list. kubectl patch deployment my-deployment -n default –patch='{ “spec”: { “template”: { “spec”: { “initContainers”: [ { “name”: “my-init-container1”, “securityContext”: { “runAsUser”: 999 } } ] } } } }’ But ‘initContainers’ is an … Kubernetes: patch every array element using kubectl and jq

Ubuntu: fixing apt NO_PUBKEY errors by converting deprecated keyring to signed-by attribute


Categories: Linux

If apt update throws warnings about invalid signature verification and NO_PUBKEY, you may need to migrate from using the deprecated system keyring to using a ‘signed-by’ attribute in your apt repo definition file. Here are examples of errors you might see when doing an ‘apt update’. W: An error occurred during the signature verification. The … Ubuntu: fixing apt NO_PUBKEY errors by converting deprecated keyring to signed-by attribute

GCP: list of available GKE cluster versions in region and channel


Categories: Hyperscaler

If you are going to create a GKE cluster in a region, you may need to be explicit with the version of the master control plane and worker nodes.  Below is how you would list the available versions. # specify your region region=us-east1 gcloud container get-server-config –region=$region

OpenWrt: upgrading to latest version when chipset migrated to DSA support


Categories: OpenWrt

Starting with OpenWrt 21, a specific list of chipsets starting using DSA (Distributed Switch Architecture) from the Linux kernel instead of swconfig.  This significantly changes the way switches and vlan are handled in OpenWrt, and therefore system configurations cannot always be migrated. Trying to run sysupgrade on a chipset that is changing from swconfig to … OpenWrt: upgrading to latest version when chipset migrated to DSA support

Bash: awk to extract Nth match from file based on line separator


Categories: Linux, Scripting

If you need to extract the Nth occurrence of a match in a file (given definitive block separators), awk provides a convenient way to express the extraction. For example, a chained pem certificate will have multiple certification definitions with unique starting and ending marker lines.  Here is how you would extract the second certificate. awk … Bash: awk to extract Nth match from file based on line separator

Jekyll: exporting a WordPress blog to a static Jekyll site on Ubuntu


Categories: Linux

If you have ever considered moving from WordPress to the Jekyll static site generator, you can preview this migration by running jekyll on your local Ubuntu host. This will allow you to assess whether you can find suitable replacements for the WordPress plugins you have come to rely upon, validate your content syntax, and tweak … Jekyll: exporting a WordPress blog to a static Jekyll site on Ubuntu

Python: TreeMap visualization of hierarchical Pandas DataFrame


Categories: Python

The Plotly graphing library has a wide array of visualizations for datasets.  And it has native support for Pandas DataFrame, which makes it convenient for datasets coming from a wide range of sources. One visualization that I find particularly useful for hierarchical data is the TreeMap, which can group on data lineage, uses area as … Python: TreeMap visualization of hierarchical Pandas DataFrame

Ubuntu: fix apt warning for Dropbox with key in legacy keyring


Categories: Linux

If you have Dropbox installed on your Linux desktop and have recently started seeing this warning message from apt: http://linux.dropbox.com/ubuntu/dists/disco/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details This can be resolved by adding the Dropbox PGP key to the ‘signed-by’ of the apt repo definition (as … Ubuntu: fix apt warning for Dropbox with key in legacy keyring

Github: automated build and publish of containerized GoLang app with Github Actions


Categories: Containers, Development

Github Actions provide the ability to define a build workflow based on Github repository events.  The workflow steps are defined as yaml and can be triggered by various events, including a code push, branch, or tagging in the repository. In this article I will detail the steps of creating a statically-linked GoLang binary that when … Github: automated build and publish of containerized GoLang app with Github Actions

Github: automated Github release of GoLang binary using Github Actions


Categories: Linux

Github Actions provide the ability to define a build workflow directly in Github.  The workflow steps are defined as yaml and can be triggered by various events, including a code push, branch, or tagging in the repository. In this article I will detail the steps of creating a statically-linked GoLang binary that is automatically built … Github: automated Github release of GoLang binary using Github Actions