DevOps

Vault: synchronizing secrets from Vault to Kubernetes using Vault Secrets Operator


Categories: Containers, DevOps, Kubernetes

The Vault Secrets Operator is a Vault integration that runs inside a Kubernetes cluster and synchronizes Vault-level secrets to Kubernetes-level secrets. This secret synchronization happens transparently to the running workloads, without any need to retrofit existing images or manifests. In this article, I will show how to: Install the Vault Secrets Operator (VSO) Configure the … Vault: synchronizing secrets from Vault to Kubernetes using Vault Secrets Operator

Vault: JWT authentication mode with multiple roles to isolate secrets


Categories: Containers, DevOps, Kubernetes

In this article, I will detail how to use Vault JWT auth mode to isolate the secrets of two different deployments in the same Kubernetes cluster.  This will be done by using two different Kubernetes Service Accounts, each of which generates unique JWT that are tied to a different Vault role. JWT auth mode is … Vault: JWT authentication mode with multiple roles to isolate secrets

Ansible: resolving error “Invalid callback for stdout specified: yaml”


Categories: DevOps

If you are getting the following error when invoking an Ansible playbook or any of the Ansible related utilities: ERROR! Invalid callback for stdout specified: yaml This means Ansible is attempting to use the new YAML callback plugin, but cannot find the Ansible Galaxy community.general module.  This module is installed by the ‘ansible’ pip module, … Ansible: resolving error “Invalid callback for stdout specified: yaml”

GitLab: Continuous Deployment with Agent for Kubernetes and GitLab pipeline


Categories: DevOps

GitLab pipelines are frequently used for the building of binaries and publishing of images to container registries, but do not always follow through with Continuous Deployment to a live environment. One reason is that pipelines do not usually have access to the internal systems where these applications are meant to be deployed. In this article, … GitLab: Continuous Deployment with Agent for Kubernetes and GitLab pipeline

GitLab: self-managed runner for CI/CD jobs on GCP VM instances


Categories: DevOps

The globally shared set of GitLab runners for CI/CD jobs works well for building binaries, publishing images, and reaching out to publicly available endpoints for services and infrastructure building. But the ability to run a private, self-managed runner can grant pipelines entirely new levels of functionality on several fronts: Can communicate openly to private, internal … GitLab: self-managed runner for CI/CD jobs on GCP VM instances

GitLab: automated build and publish of multi-platform container image with GitLab pipeline


Categories: Containers, DevOps

GitLab CI/CD pipelines can be used to automatically build and push Docker images to the GitLab Container Registry. Beyond building a simple image, in this article I will show how to define a workflow that builds and pushes a multi-platform image (amd64,arm64,arm32) with manifest index to the GitLab Container Registry.  This is enabled by using … GitLab: automated build and publish of multi-platform container image with GitLab pipeline

Helm: automated publishing of Helm repo with Github Actions


Categories: DevOps, Kubernetes

In a previous article, I described how to expose a Github source repo as a public Helm repository by enabling Github Pages and running the chart-releaser utility. In this article, I want to remove the manual invocation of the chart-releaser, and instead place that into an Github Actions workflow that automatically publishes changes to the … Helm: automated publishing of Helm repo with Github Actions

Helm: manually publishing Helm repo on Github using chart-releaser


Categories: DevOps, Kubernetes

The only requirement for a public Helm chart repository is that it exposes a URL named “index.yaml”.   So by adding a file named “index.yaml” to source control and enabling Github Pages to serve the file over HTTPS, you have the minimal basis for a public Helm chart repository. The backing Chart content (.tgz) can also … Helm: manually publishing Helm repo on Github using chart-releaser

Github: automated build and publish of multi-platform container image with Github Actions


Categories: Containers, DevOps

Github Actions provide the ability to define a build workflow based on Github repository events.  The workflow steps are defined as yaml and can be triggered by various events, including a code push, branch, or tagging in the repository. In this article, I will show how to define workflow steps that build and push a … Github: automated build and publish of multi-platform container image with Github Actions

Helm: discovering Helm chart releases installed into Kubernetes cluster


Categories: DevOps, Kubernetes

If you are administering a Kubernetes cluster that you have inherited or perhaps not visited in a while, then you may need to reacquaint yourself with: which Helm charts are installed into what namespaces, if there are chart updates available, and then what values were used for chart installation. Below are commands that can assist … Helm: discovering Helm chart releases installed into Kubernetes cluster

Terraform: error removing module containing legacy provider block, ‘Provider configuration not present’


Categories: DevOps

If you have just removed a module declaration from your Terraform configuration and now get a ‘Provider configuration not present’ error when running apply: Error: Provider configuration not present To work with module.mymodule_legacysyntax.null_resource.test_rs (orphan) its original provider configuration at module.mymodule_legacysyntax.provider[“registry.terraform.io/hashicorp/null”] is required, but it has been removed. This occurs when a provider configuration is removed … Terraform: error removing module containing legacy provider block, ‘Provider configuration not present’

Ansible: resolving ‘could not initialize the preferred locale: unsupported locale setting’


Categories: DevOps

If you are getting the following error when invoking ‘ansible’, ‘ansible-playbook’, ‘ansible-galaxy’ or any of the Ansible related utilities: ERROR: Ansible could not initialize the preferred locale: unsupported locale setting This means Ansible cannot find a locale ending in “.UTF-8”.  Check the currently installed locales: $ locale -a Then export the LC_ALL variable to one … Ansible: resolving ‘could not initialize the preferred locale: unsupported locale setting’

Terraform: terraform_remote_state to pass values to other configurations


Categories: DevOps, Kubernetes

It would be uncommon to have one monolithic Terraform configuration for all the infrastructure in your organization.  More than likely, there are multiple groups and each has responsibility and ownership of certain components (e.g. networking, storage, authorization, Kubernetes). As an example, let’s say your responsibility is the Kubernetes cluster build. You may need the following … Terraform: terraform_remote_state to pass values to other configurations

Terraform: fixing error “querying Cloud Storage failed: storage: bucket doesn’t exist”


Categories: DevOps

If you are attempting to run “terraform init” with a Google Cloud Storage backend and get the following error: Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn’t exist The first check should be that the Google Cloud Storage bucket indeed exists, using gsutil. project_id=myproject-123 gsutil ls -p $project_id If the … Terraform: fixing error “querying Cloud Storage failed: storage: bucket doesn’t exist”

GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster


Categories: Automation, DevOps, Hyperscaler

As much as Terraform pushes to be the absolute system of record for resources it creates, often valid external processes are assisting in managing those same resources. Here are some examples of legitimate external changes: Other company-approved Terraform scripts applying labeling to resources in order to track ownership and costs Security teams modifying IAM roles … GKE: terraform lifecycle ‘ignore_changes’ to manage external changes to GKE cluster

Terraform: migrate state from local to remote Google Cloud Storage bucket and back


Categories: DevOps, Hyperscaler

In this article I will demonstrate how to take a Terraform configuration that is using a local state file and migrate its persistent state to a remote Google Cloud Storage bucket (GCS).  We will then perform the migration again, but this time to bring the remote state back to a local file. We will illustrate … Terraform: migrate state from local to remote Google Cloud Storage bucket and back

Helm: Installing Helm on Ubuntu


Categories: Containers, DevOps

Update Aug 2023: using newer ‘signed-by’ attribute for apt signing keys. Installing Helm using apt is a straight-forward procedure and documented on the official site.  Coming straight from the official helm documentation, here are the commands for Ubuntu 22. curl https://baltocdn.com/helm/signing.asc | gpg –dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null sudo chmod 644 /usr/share/keyrings/helm.gpg sudo … Helm: Installing Helm on Ubuntu

RabbitMQ: Deleting a ghost queue that cannot be removed at the GUI/CLI


Categories: DevOps

If you get a timeout/errors trying to delete a RabbitMQ queue from  the management dashboard or CLI with an error similar in syntax to below: failed to perform operation on queue ” in vhost ” due to timeout Then you can attempt deletion using rabbitmqctl to evaluate an Erlang expression: rabbitmqctl eval ‘Q = {resource, … RabbitMQ: Deleting a ghost queue that cannot be removed at the GUI/CLI

SaltStack: salt-ssh for agentless automation on Ubuntu


Categories: Automation, DevOps

Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale.  Even in the growing world of containerization, there is the need for bulk automation. This article will detail installation of  Salt SSH which leverages the power of SaltStack without the requirements for an agent install.

SaltStack: Installing a Salt Master on Ubuntu Xenial


Categories: Automation, DevOps

Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale.  Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion. This article will detail installation of the SaltStack master on Ubuntu Xenial 16.04, with validation using a single Minion. Note that … SaltStack: Installing a Salt Master on Ubuntu Xenial

Maven: Installing a private Maven repository on Ubuntu using Apache Archiva


Categories: DevOps, Java

An essential part of the standard build process for Java applications is having a repository where project artifacts are stored. Artifact curation provides the ability to manage dependencies, quickly rollback releases, support compatibility of downstream projects, do QA promotion from test to production, support a continuous build pipeline, and provides auditability. Archiva from the Apache … Maven: Installing a private Maven repository on Ubuntu using Apache Archiva

SaltStack: Installing a Salt Master on Ubuntu 14.04


Categories: DevOps, Linux

Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale.  Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion. This article will detail installation of the SaltStack master on Ubuntu 14.04, with validation using a single Minion.  Note that Minion … SaltStack: Installing a Salt Master on Ubuntu 14.04

ELK: Installing MetricBeat for collecting system and application metrics


Categories: DevOps, Monitoring

ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host.  That means that along with standard cpu/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language. In this article we will describe installing … ELK: Installing MetricBeat for collecting system and application metrics

ELK: ElastAlert for alerting based on data from ElasticSearch


Categories: DevOps, Linux, Logging, Monitoring

ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert. ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.

ELK: Using Curator to manage the size and persistence of your index storage


Categories: DevOps

The Curator product from ElasticSearch allows you to apply batch actions to your indexes (close, create, delete, etc.).  One specific use case is applying a retention policy to your indexes, deleting any indexes that are older than a certain threshold. Installation Start by installing Curator using apt and pip: $ sudo apt-get install python-pip -y … ELK: Using Curator to manage the size and persistence of your index storage

VirtualBox: Installing VirtualBox and Vagrant on Ubuntu 14.04/16.04


Categories: DevOps

Although container based engines such as Docker are highly popularized for newer application deployment – there will still be widespread use of OS virtualization engines for years to come. One of the most popular virtualization engines for development purposes is the open-source VirtualBox from Oracle.  This article will detail its installation on Ubuntu 14.04.

Docker: Sending Spring Boot logging to syslog


Categories: Containers, DevOps, Java, Monitoring

Building services using Spring Boot gives a development team a jump start on many production concerns, including logging.  But unlike a standard deployment where logging to a local file is where the developer’s responsibility typically ends, with Docker we must think about how to log to a public space outside our ephemeral container space. The … Docker: Sending Spring Boot logging to syslog

Squid: Configuring an Ubuntu host to use a Squid proxy for internet access


Categories: DevOps, Linux

Once you have a Squid proxy setup as described in my article here, the next challenge is configuring your Ubuntu servers so that they use this proxy by default instead of attempting direct internet connections. There are several entities we want using Squid by default: apt package manager, interactive consoles and wget/curl, and Java applications.

HAProxy: Using HAProxy for SSL termination on Ubuntu


Categories: DevOps, Linux

HAProxy is a high performance TCP/HTTP (Level 4 and Level 7) load balancer and reverse proxy.  A common pattern is allowing HAProxy to be the fronting SSL-termination point, and then HAProxy determines which pooled backend server serves the request.

Nginx: Using Nginx for SSL termination on Ubuntu


Categories: DevOps, Linux

Nginx is a popular reverse proxy and load balancer that focuses on level 7 (application) traffic.  A common pattern is allowing Nginx to be the fronting SSL-termination point, and then Nginx determines which pooled backend server is best available to serve the request.

Apache2: Enable LDAP authentication and SSL termination for Ubuntu


Categories: DevOps, Linux

Some web applications leave authentication as an orthogonal concern to the application – not including any kind of login functionality and instead leaving authentication as an operational concern. When this happens, a reverse proxy that has an LDAP integration can act as an architectural sentry in front of the web application and also fulfills the … Apache2: Enable LDAP authentication and SSL termination for Ubuntu

Jenkins: Setting up a continuous integration server on Ubuntu


Categories: DevOps, Linux

Jenkins is the open-source automation server that is critical in building a continuous integration and delivery pipeline.  It is extensible and has a wealth of plugins that  integrate with numerous enterprise systems. Here are the detailed steps for installing a Jenkins server on Ubuntu.

Maven: Installing a 3rd party jar to a local or remote repository


Categories: DevOps, Java

Especially in enterprise application development, there can be 3rd party dependencies that are not available in public Maven repositories.  These may be internal, business specific libraries or licensed libraries that have limitations on usage. When this is the case, you can either publish to a private Maven repository that controls authorization or you can put … Maven: Installing a 3rd party jar to a local or remote repository

Maven: Installing a private Maven repository on Ubuntu using Artifactory


Categories: DevOps, Java

An essential part of the standard build process for Java applications is having a set of repositories where project artifacts are stored. Artifact curation provides the ability to manage dependencies, quickly rollback releases, support compatibility of downstream projects, do QA promotion from test to production, support a continuous build pipeline, and provides auditability. JFrog puts … Maven: Installing a private Maven repository on Ubuntu using Artifactory