GitLab: Continuous Deployment with Agent for Kubernetes and GitLab pipeline


Categories: DevOps

GitLab pipelines are frequently used for the building of binaries and publishing of images to container registries, but do not always follow through with Continuous Deployment to a live environment. One reason is that pipelines do not usually have access to the internal systems where these applications are meant to be deployed. In this article, … GitLab: Continuous Deployment with Agent for Kubernetes and GitLab pipeline

GitLab: self-managed runner for CI/CD jobs on GCP VM instances


Categories: DevOps

The globally shared set of GitLab runners for CI/CD jobs works well for building binaries, publishing images, and reaching out to publicly available endpoints for services and infrastructure building. But the ability to run a private, self-managed runner can grant pipelines entirely new levels of functionality on several fronts: Can communicate openly to private, internal … GitLab: self-managed runner for CI/CD jobs on GCP VM instances

Bash: extracting first or last N octets, paths, or domain from string with fixed separator


Categories: Linux

When parsing a string that is divided by a separator char, getting the first N values OR last N values is a common scenario when dealing with: IP address separated by periods, e.g. “10.11.12.13” File path separated by forward slash “/tmp/myfolder/subpath1/subpath2/subpath3” Fully qualified domain separated by periods “sub1.sub2.my.domain.com”

GitLab: automated build and publish of multi-platform container image with GitLab pipeline


Categories: Containers, DevOps

GitLab CI/CD pipelines can be used to automatically build and push Docker images to the GitLab Container Registry. Beyond building a simple image, in this article I will show how to define a workflow that builds and pushes a multi-platform image (amd64,arm64,arm32) with manifest index to the GitLab Container Registry.  This is enabled by using … GitLab: automated build and publish of multi-platform container image with GitLab pipeline

Helm: automated publishing of Helm repo with Github Actions


Categories: DevOps, Kubernetes

In a previous article, I described how to expose a Github source repo as a public Helm repository by enabling Github Pages and running the chart-releaser utility. In this article, I want to remove the manual invocation of the chart-releaser, and instead place that into an Github Actions workflow that automatically publishes changes to the … Helm: automated publishing of Helm repo with Github Actions

Helm: manually publishing Helm repo on Github using chart-releaser


Categories: DevOps, Kubernetes

The only requirement for a public Helm chart repository is that it exposes a URL named “index.yaml”.   So by adding a file named “index.yaml” to source control and enabling Github Pages to serve the file over HTTPS, you have the minimal basis for a public Helm chart repository. The backing Chart content (.tgz) can also … Helm: manually publishing Helm repo on Github using chart-releaser

Github: automated build and publish of multi-platform container image with Github Actions


Categories: Containers, DevOps

Github Actions provide the ability to define a build workflow based on Github repository events.  The workflow steps are defined as yaml and can be triggered by various events, including a code push, branch, or tagging in the repository. In this article, I will show how to define workflow steps that build and push a … Github: automated build and publish of multi-platform container image with Github Actions

Docker: building multi-platform images that use fat manifest list/index


Categories: Containers, Development

Docker can build multi-platform images that use a manifest index (fat manifest list) by using the Docker buildx command with backing containerd runtime and QEMU for cross-platform emulation. Using a manifest index for multi-platform images simplifies application level orchestration by using the same name and version for all architectures.  For example: # same image name … Docker: building multi-platform images that use fat manifest list/index

Docker: installing Docker CE on Ubuntu


Categories: Containers, Linux

Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization. These instructions are taken from the official Docker for Ubuntu page, but I fine-tuned them per Ubuntu22+ standards.

Helm: discovering Helm chart releases installed into Kubernetes cluster


Categories: DevOps, Kubernetes

If you are administering a Kubernetes cluster that you have inherited or perhaps not visited in a while, then you may need to reacquaint yourself with: which Helm charts are installed into what namespaces, if there are chart updates available, and then what values were used for chart installation. Below are commands that can assist … Helm: discovering Helm chart releases installed into Kubernetes cluster

Kubernetes: patching container arguments array with kubectl and jq


Categories: Kubernetes

The need to configure a specific pod’s container arguments is a common Kubernetes administration task.  As examples, you might need to enable verbose logging, set an explicit value to override a default, or configure a host name or port set in a container’s arguments. In the example below, we are targeting the ‘metrics-server’ in the … Kubernetes: patching container arguments array with kubectl and jq

Kubernetes: HorizontalPodAutoscaler evaluation based on Prometheus metric


Categories: Kubernetes, Monitoring

HorizontalPodAutoscaler (HPA) allow you to dynamically scale the replica count of your Deployment based on basic CPU/memory resource metrics from the metrics-server.  If you want scaling based on more advanced scenarios and you are already using the Prometheus stack, the prometheus-adapter provides this enhancement. The prometheus-adapter takes basic Prometheus metrics, and then synthesizes custom API … Kubernetes: HorizontalPodAutoscaler evaluation based on Prometheus metric

Kubernetes: implementing and testing a HorizontalPodAutoscaler


Categories: Kubernetes

HorizontalPodAutoscaler (HPA) allow you to dynamically scale the replica count of your Deployment based on criteria such as memory or CPU utilization, which make it great way to manage spikes in utilization while still keeping your cluster size and infrastructure costs managed effectively. In order for HPA to evaluate CPU and memory utilization and take … Kubernetes: implementing and testing a HorizontalPodAutoscaler

Kubernetes: fixing x509 certificate errors from metric-server on K3s cluster


Categories: Kubernetes

K3s is deployed by default with a metrics-server, but if you have a multi-node cluster it will fail unless you add the names of all the nodes to the kube-apiserver certificate.  Symptoms of this problem include: metrics-server deployment will throw x509 errors in its log Error when you try to run “kubectl top pods” No … Kubernetes: fixing x509 certificate errors from metric-server on K3s cluster

Bash: decoding a JWT from the command line with jq


Categories: Linux

Although jwt.io has become a common online destination for decoding JWT, this can also be done locally using jq. # populate JWT variable JWT=… # decode with jq utility echo $JWT | jq -R ‘split(“.”) | .[0],.[1] | @base64d | fromjson’ Attribution of credit goes to this gist.

Terraform: error removing module containing legacy provider block, ‘Provider configuration not present’


Categories: DevOps

If you have just removed a module declaration from your Terraform configuration and now get a ‘Provider configuration not present’ error when running apply: Error: Provider configuration not present To work with module.mymodule_legacysyntax.null_resource.test_rs (orphan) its original provider configuration at module.mymodule_legacysyntax.provider[“registry.terraform.io/hashicorp/null”] is required, but it has been removed. This occurs when a provider configuration is removed … Terraform: error removing module containing legacy provider block, ‘Provider configuration not present’

Ansible: resolving ‘could not initialize the preferred locale: unsupported locale setting’


Categories: DevOps

If you are getting the following error when invoking ‘ansible’, ‘ansible-playbook’, ‘ansible-galaxy’ or any of the Ansible related utilities: ERROR: Ansible could not initialize the preferred locale: unsupported locale setting This means Ansible cannot find a locale ending in “.UTF-8”.  Check the currently installed locales: $ locale -a Then export the LC_ALL variable to one … Ansible: resolving ‘could not initialize the preferred locale: unsupported locale setting’

Kubernetes: evaluating full readiness of deployment, daemonset, or pod


Categories: Kubernetes

Deployments and Daemonset typically have more than one replica or desired replica count, and although kubectl default formatting will return columns summarizing how many are desired and how many are currently ready, an automated script needs to parse these value in order to determine if full health. Similiarly, pod status as well as the readiness … Kubernetes: evaluating full readiness of deployment, daemonset, or pod

Terraform: terraform_remote_state to pass values to other configurations


Categories: DevOps, Kubernetes

It would be uncommon to have one monolithic Terraform configuration for all the infrastructure in your organization.  More than likely, there are multiple groups and each has responsibility and ownership of certain components (e.g. networking, storage, authorization, Kubernetes). As an example, let’s say your responsibility is the Kubernetes cluster build. You may need the following … Terraform: terraform_remote_state to pass values to other configurations

Kubernetes: creating TLS secrets with kustomize using embedded or external content


Categories: Kubernetes

There are multiple options for creating a TLS secret using kustomize.  One is to embed the certificate content as a base64 string directly in the data, the other is to use an external file. Below is an example kustomization.yaml file that serves as an entry point for both methods. — apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: … Kubernetes: creating TLS secrets with kustomize using embedded or external content

Terraform: fixing error “querying Cloud Storage failed: storage: bucket doesn’t exist”


Categories: DevOps

If you are attempting to run “terraform init” with a Google Cloud Storage backend and get the following error: Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn’t exist The first check should be that the Google Cloud Storage bucket indeed exists, using gsutil. project_id=myproject-123 gsutil ls -p $project_id If the … Terraform: fixing error “querying Cloud Storage failed: storage: bucket doesn’t exist”

GitLab: generating URL that can be used for Merge Request from fork to upstream


Categories: Development

The forked workflow is popularized by the Open Source community where your personal contributions are made by having your own personal fork of a repository and pushing a GitLab Merge Request to a central repository. A GitLab Merge Request can be submitted from the web UI by clicking on “Merge requests” and manually selecting the … GitLab: generating URL that can be used for Merge Request from fork to upstream

GCP: determining whether ASM is installed via asmcli or gcloud fleet


Categories: Hyperscaler

Anthos Service Mesh for GKE can be installed in the following modes: In-cluster ASM using the asmcli utility Managed ASM using the asmcli utility Managed ASM using the ‘gcloud container fleet’ command Managed ASM using the Terraform asm submodule If you need to determine the installation mode used on your GKE cluster, you can examine … GCP: determining whether ASM is installed via asmcli or gcloud fleet

Bash: testing if a file exists, has content, and is recently modified


Categories: Linux

If you need to test for a file’s existence, content size, and whether it was recently modified, the ‘find‘ utility can provide this functionality in a single call. One scenario for this usage might be the cached results from a remote service call (database, REST service, etc).  If fetching these results was a relatively costly … Bash: testing if a file exists, has content, and is recently modified

GCP: determining whether GKE cluster mode is Standard or Autopilot


Categories: Hyperscaler

If you need to determine at the CLI whether a GKE cluster is managed using Standard or Autopilot mode, this is available by using gcloud to describe the cluster. # identify cluster and location gcloud container clusters list cluster_name=<clusterName> location_flag=”–region=<region>” # OR –zone=<zone> # returns ‘True’ if GKE AutoPilot cluster # returns empty if standard … GCP: determining whether GKE cluster mode is Standard or Autopilot

GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster


Categories: Automation, DevOps, Hyperscaler

As much as Terraform pushes to be the absolute system of record for resources it creates, often valid external processes are assisting in managing those same resources. Here are some examples of legitimate external changes: Other company-approved Terraform scripts applying labeling to resources in order to track ownership and costs Security teams modifying IAM roles … GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster

GCP: Cloud Run with build trigger coming from remote GitHub repository


Categories: Containers, Development, Hyperscaler

GCP build triggers can easily handle Continuous Deployment (CD) when the source code is homed in a Google Cloud Source repository.  But even if the system of record for your source is a remote GitHub repository, these same type of push and tag events can be consumed if you configure a connection and repository link. … GCP: Cloud Run with build trigger coming from remote GitHub repository

GCP: Cloud Run/Function to handle requests to GKE cluster during maintenance


Categories: Hyperscaler

At some point, there will be a system change significant enough that a maintenance window needs to be scheduled with customers.   But that doesn’t mean the end-user traffic or client integrations will stop requesting the services. What we need to present to end-users is a maintenance page during this outage to indicate the overall solution … GCP: Cloud Run/Function to handle requests to GKE cluster during maintenance

Ansible: adding custom apt repository with ‘signed-by’ gpg key


Categories: Automation

The centralized system keyring for apt was deprecated starting in Ubuntu 21, and is being replaced with an explicit path to the local gpg key in the ‘signed-by’ attribute. I have written more extensive articles on this subject [here,here], but from an Ansible perspective, this means ensuring the gpg key is downloaded to ‘/usr/share/keyrings’ with … Ansible: adding custom apt repository with ‘signed-by’ gpg key