Vault: Spring Boot web app using Spring Cloud Vault to fetch secrets

HashiCorp Vault is a secret and encryption management system that allows your organization to secure sensitive information such as API keys, certificates, and passwords. In this article, I will show how a Java Spring Boot web application deployed into a Kubernetes cluster can fetch a secret directly from the Vault server using the Spring Cloud Vault: Spring Boot web app using Spring Cloud Vault to fetch secrets

Vault: HashiCorp Vault deployed into Kubernetes cluster for secret management

HashiCorp Vault is a secret and encryption management system that allows your organization to secure sensitive information such as API keys, certificates, and passwords. It has tight integrations with Kubernetes that allows containers to fetch secrets without requiring hardcoding them into environment variables, files, or external services. The official docs already provide usage scenarios, so Vault: HashiCorp Vault deployed into Kubernetes cluster for secret management

Bash: fixing SSH authentication error “bad ownership or modes for file/directory”

If ssh private/public keypair authentication is failing, check the logs on the server side for permission errors.  On Debian/Ubuntu check for these errors in “/var/log/auth.log”. # error if authorized_keys file has too wide a permission for others Authentication refused: bad ownership or modes for file /home/myuser/.ssh/authorized_keys # error if .ssh directory has too wide a Bash: fixing SSH authentication error “bad ownership or modes for file/directory”

GitLab: Continuous Deployment with Agent for Kubernetes and GitLab pipeline

GitLab pipelines are frequently used for the building of binaries and publishing of images to container registries, but do not always follow through with Continuous Deployment to a live environment. One reason is that pipelines do not usually have access to the internal systems where these applications are meant to be deployed. In this article, GitLab: Continuous Deployment with Agent for Kubernetes and GitLab pipeline

GitLab: self-managed runner for CI/CD jobs on GCP VM instances

The globally shared set of GitLab runners for CI/CD jobs works well for building binaries, publishing images, and reaching out to publicly available endpoints for services and infrastructure building. But the ability to run a private, self-managed runner can grant pipelines entirely new levels of functionality on several fronts: Can communicate openly to private, internal GitLab: self-managed runner for CI/CD jobs on GCP VM instances

Bash: extracting first or last N octets, paths, or domain from string with fixed separator

When parsing a string that is divided by a separator char, getting the first N values OR last N values is a common scenario when dealing with: IP address separated by periods, e.g. “10.11.12.13” File path separated by forward slash “/tmp/myfolder/subpath1/subpath2/subpath3” Fully qualified domain separated by periods “sub1.sub2.my.domain.com”

GitLab: automated build and publish of multi-platform container image with GitLab pipeline

GitLab CI/CD pipelines can be used to automatically build and push Docker images to the GitLab Container Registry. Beyond building a simple image, in this article I will show how to define a workflow that builds and pushes a multi-platform image (amd64,arm64,arm32) with manifest index to the GitLab Container Registry.  This is enabled by using GitLab: automated build and publish of multi-platform container image with GitLab pipeline

Helm: automated publishing of Helm repo with Github Actions

In a previous article, I described how to expose a Github source repo as a public Helm repository by enabling Github Pages and running the chart-releaser utility. In this article, I want to remove the manual invocation of the chart-releaser, and instead place that into an Github Actions workflow that automatically publishes changes to the Helm: automated publishing of Helm repo with Github Actions

Helm: manually publishing Helm repo on Github using chart-releaser

The only requirement for a public Helm chart repository is that it exposes a URL named “index.yaml”.   So by adding a file named “index.yaml” to source control and enabling Github Pages to serve the file over HTTPS, you have the minimal basis for a public Helm chart repository. The backing Chart content (.tgz) can also Helm: manually publishing Helm repo on Github using chart-releaser

Github: automated build and publish of multi-platform container image with Github Actions

Github Actions provide the ability to define a build workflow based on Github repository events.  The workflow steps are defined as yaml and can be triggered by various events, including a code push, branch, or tagging in the repository. In this article, I will show how to define workflow steps that build and push a Github: automated build and publish of multi-platform container image with Github Actions

Docker: building multi-platform images that use fat manifest list/index

Docker can build multi-platform images that use a manifest index (fat manifest list) by using the Docker buildx command with backing containerd runtime and QEMU for cross-platform emulation. Using a manifest index for multi-platform images simplifies application level orchestration by using the same name and version for all architectures.  For example: # same image name Docker: building multi-platform images that use fat manifest list/index

Docker: installing Docker CE on Ubuntu

Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization. These instructions are taken from the official Docker for Ubuntu page, but I fine-tuned them per Ubuntu22+ standards.

Helm: discovering Helm chart releases installed into Kubernetes cluster

If you are administering a Kubernetes cluster that you have inherited or perhaps not visited in a while, then you may need to reacquaint yourself with: which Helm charts are installed into what namespaces, if there are chart updates available, and then what values were used for chart installation. Below are commands that can assist Helm: discovering Helm chart releases installed into Kubernetes cluster

Kubernetes: patching container arguments array with kubectl and jq

The need to configure a specific pod’s container arguments is a common Kubernetes administration task.  As examples, you might need to enable verbose logging, set an explicit value to override a default, or configure a host name or port set in a container’s arguments. In the example below, we are targeting the ‘metrics-server’ in the Kubernetes: patching container arguments array with kubectl and jq

Kubernetes: HorizontalPodAutoscaler evaluation based on Prometheus metric

HorizontalPodAutoscaler (HPA) allow you to dynamically scale the replica count of your Deployment based on basic CPU/memory resource metrics from the metrics-server.  If you want scaling based on more advanced scenarios and you are already using the Prometheus stack, the prometheus-adapter provides this enhancement. The prometheus-adapter takes basic Prometheus metrics, and then synthesizes custom API Kubernetes: HorizontalPodAutoscaler evaluation based on Prometheus metric

Kubernetes: implementing and testing a HorizontalPodAutoscaler

HorizontalPodAutoscaler (HPA) allow you to dynamically scale the replica count of your Deployment based on criteria such as memory or CPU utilization, which make it great way to manage spikes in utilization while still keeping your cluster size and infrastructure costs managed effectively. In order for HPA to evaluate CPU and memory utilization and take Kubernetes: implementing and testing a HorizontalPodAutoscaler

Kubernetes: fixing x509 certificate errors from metric-server on K3s cluster

K3s is deployed by default with a metrics-server, but if you have a multi-node cluster it will fail unless you add the names of all the nodes to the kube-apiserver certificate.  Symptoms of this problem include: metrics-server deployment will throw x509 errors in its log Error when you try to run “kubectl top pods” No Kubernetes: fixing x509 certificate errors from metric-server on K3s cluster

Bash: decoding a JWT from the command line with jq

Although jwt.io has become a common online destination for decoding JWT, this can also be done locally using jq. # populate JWT variable JWT=… # decode with jq utility echo $JWT | jq -R ‘split(“.”) | .[0],.[1] | @base64d | fromjson’ Attribution of credit goes to this gist.

Terraform: error removing module containing legacy provider block, ‘Provider configuration not present’

If you have just removed a module declaration from your Terraform configuration and now get a ‘Provider configuration not present’ error when running apply: Error: Provider configuration not present To work with module.mymodule_legacysyntax.null_resource.test_rs (orphan) its original provider configuration at module.mymodule_legacysyntax.provider[“registry.terraform.io/hashicorp/null”] is required, but it has been removed. This occurs when a provider configuration is removed Terraform: error removing module containing legacy provider block, ‘Provider configuration not present’

Ansible: resolving ‘could not initialize the preferred locale: unsupported locale setting’

If you are getting the following error when invoking ‘ansible’, ‘ansible-playbook’, ‘ansible-galaxy’ or any of the Ansible related utilities: ERROR: Ansible could not initialize the preferred locale: unsupported locale setting This means Ansible cannot find a locale ending in “.UTF-8”.  Check the currently installed locales: $ locale -a Then export the LC_ALL variable to one Ansible: resolving ‘could not initialize the preferred locale: unsupported locale setting’

Kubernetes: evaluating full readiness of deployment, daemonset, or pod

Deployments and Daemonset typically have more than one replica or desired replica count, and although kubectl default formatting will return columns summarizing how many are desired and how many are currently ready, an automated script needs to parse these value in order to determine if full health. Similiarly, pod status as well as the readiness Kubernetes: evaluating full readiness of deployment, daemonset, or pod

Terraform: terraform_remote_state to pass values to other configurations

It would be uncommon to have one monolithic Terraform configuration for all the infrastructure in your organization.  More than likely, there are multiple groups and each has responsibility and ownership of certain components (e.g. networking, storage, authorization, Kubernetes). As an example, let’s say your responsibility is the Kubernetes cluster build. You may need the following Terraform: terraform_remote_state to pass values to other configurations

Kubernetes: creating TLS secrets with kustomize using embedded or external content

There are multiple options for creating a TLS secret using kustomize.  One is to embed the certificate content as a base64 string directly in the data, the other is to use an external file. Below is an example kustomization.yaml file that serves as an entry point for both methods. — apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: Kubernetes: creating TLS secrets with kustomize using embedded or external content

Terraform: fixing error “querying Cloud Storage failed: storage: bucket doesn’t exist”

If you are attempting to run “terraform init” with a Google Cloud Storage backend and get the following error: Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn’t exist The first check should be that the Google Cloud Storage bucket indeed exists, using gsutil. project_id=myproject-123 gsutil ls -p $project_id If the Terraform: fixing error “querying Cloud Storage failed: storage: bucket doesn’t exist”

GitLab: generating URL that can be used for Merge Request from fork to upstream

The forked workflow is popularized by the Open Source community where your personal contributions are made by having your own personal fork of a repository and pushing a GitLab Merge Request to a central repository. A GitLab Merge Request can be submitted from the web UI by clicking on “Merge requests” and manually selecting the GitLab: generating URL that can be used for Merge Request from fork to upstream

GCP: determining whether ASM is installed via asmcli or gcloud fleet

Anthos Service Mesh for GKE can be installed in the following modes: In-cluster ASM using the asmcli utility Managed ASM using the asmcli utility Managed ASM using the ‘gcloud container fleet’ command Managed ASM using the Terraform asm submodule If you need to determine the installation mode used on your GKE cluster, you can examine GCP: determining whether ASM is installed via asmcli or gcloud fleet

Bash: testing if a file exists, has content, and is recently modified

If you need to test for a file’s existence, content size, and whether it was recently modified, the ‘find‘ utility can provide this functionality in a single call. One scenario for this usage might be the cached results from a remote service call (database, REST service, etc).  If fetching these results was a relatively costly Bash: testing if a file exists, has content, and is recently modified

GCP: determining whether GKE cluster mode is Standard or Autopilot

If you need to determine at the CLI whether a GKE cluster is managed using Standard or Autopilot mode, this is available by using gcloud to describe the cluster. # identify cluster and location gcloud container clusters list cluster_name=<clusterName> location_flag=”–region=<region>” # OR –zone=<zone> # returns ‘True’ if GKE AutoPilot cluster # returns empty if standard GCP: determining whether GKE cluster mode is Standard or Autopilot

GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster

As much as Terraform pushes to be the absolute system of record for resources it creates, often valid external processes are assisting in managing those same resources. Here are some examples of legitimate external changes: Other company-approved Terraform scripts applying labeling to resources in order to track ownership and costs Security teams modifying IAM roles GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster