The default setting for pod DNS resolution has CoreDNS use the settings from the underlying OS of the worker node. If your Kubernetes VMs are joined to multiple networks or search domains, this can cause unexpected results as well as performance issues. If you are using K3s, you can provide an independent resolv.conf file for … Kubernetes: independent resolv.conf for CoreDNS with K3s
The default setting for pod DNS resolution has CoreDNS use the settings from the underlying OS of the worker node. If your Kubernetes VMs are joined to multiple networks or search domains, this can cause unexpected results as well as performance issues. If you are using kubeadm, you can provide an independent resolv.conf file for … Kubernetes: independent resolv.conf for CoreDNS with kubeadm
The kube-prometheus-stack bundles the Prometheus Operator, monitors/rules, Grafana dashboards, and AlertManager needed to monitor a Kubernetes cluster. But there are customizations necessary to tailor the Helm installation for a Kubernetes cluster built using kubeadm. In this article, I will detail the necessary modifications to deploy a healthy monitoring stack on a kubeadm cluster.
If you are running the Prometheus Operator (e.g. with kube-prometheus-stack) then you can specify additional scrape config jobs to monitor your custom services. An additional scrape config uses regex evaluation to find matching services en masse, and targets a set of services based on label, annotation, namespace, or name. Note that adding an additional scrape … Prometheus: monitoring services using additional scrape config for Prometheus Operator
If you are running the Prometheus Operator as part of your monitoring stack (e.g. kube-prometheus-stack) then you can have your custom Service monitored by defining a ServiceMonitor CRD. The ServiceMonitor is an object that defines the service endpoints that should be scraped by Prometheus and at what interval. In this article, we will deploy a … Prometheus: monitoring a custom Service using ServiceMonitor and PrometheusRule
If your Grafana deployment is using a sidecar to watch for new dashboards defined as a ConfigMap, then adding a dashboard is a dynamic operation that can be done without even restarting the pod. If you have deployed the Prometheus/Grafana stack with kube-prometheus-stack, then you can check for the existence of the ‘grafana-sc-dashboard’ sidecar using: … Prometheus: adding a Grafana dashboard using a ConfigMap
If you need to validate your AlertManager routing configuration by sending a test alert through AlertManager, you can port-forward the AlertManager pod and send it from curl on your development host. Real alerts typically have scrape delays and then durations that must be met, so this is a way of getting almost immediate feedback on … Prometheus: sending a test alert through AlertManager
While working on your Spring Boot web application locally, gradle provides the ‘bootRun’ for a quick development lifecycle and ‘bootJar’ for packaging all the dependencies as a single jar deliverable. But for most applications these days, you will need this packaged into an OCI compatible (i.e. Docker) image for its ultimate deployment to an orchestrator … Java: build OCI compatible image for Spring Boot web app using jib
The kube-prometheus-stack bundles AlertManager for taking action on Prometheus alerts. And if you are customizing the Heml custom values file to configure email alerting, there are multiple options available. The simplest is to allow the system to fallback to using the default subject and html templates. But if you need to tailor the email content … Prometheus: external template for AlertManager html email with kube-prometheus-stack
The kube-prometheus-stack bundles Prometheus, Grafana, and AlertManager for monitoring a Kubernetes cluster. By default, the Ingress of these services is disabled. In this article I will show you how to expose these services with NGINX Ingress either via subdomain (e.g. prometheus.my.domain) or web context (e.g. my.domain/prometheus). You would not want to expose these monitoring applications … Prometheus: exposing Prometheus/Grafana as Ingress for kube-prometheus-stack
The kube-prometheus-stack bundles the Prometheus Operator, monitors/rules, Grafana dashboards, and AlertManager needed to monitor a Kubernetes cluster. But there are customizations necessary to tailor the Helm installation for K3s, a lightweight Kubernetes installation. In this article, I will detail the necessary modifications to deploy a healthy monitoring stack on a K3s cluster.
If you have a Kubernetes yaml manifest that contains multiple documents, targeting a single document for modification while still outputting the other documents untouched can be a challenge. As an example, consider the simple example below were you have a single yaml file that contains: a Namespace, Deployment, and DaemonSet. And we want to add … Kubernetes: targeting the addition of array items to a multi-document yaml manifest
Spring Boot excels as a framework that makes it convenient to expose a set of REST based services. But once an application is developed, it is so trivial to create a new resource or modify a method signature that it becomes difficult to keep the documentation up-to-date so that clients can properly consume the REST … Java: Spring Boot REST service with OpenAPI/Swagger documentation
A Kubernetes liveness and readiness probe is how the kubelet determines health of a pod. This is often times as simple as checking the ability to reach the main service port over TCP or HTTP. But if you are using Spring Boot and have enabled the Actuator dependency, you have the ability to create even … Kubernetes: liveness probe for Spring Boot with custom Actuator health check
While working on your Spring Boot web application locally, gradle provides the ‘bootRun’ for a quick development lifecycle and ‘bootJar’ for packaging all the dependencies as a single jar deliverable. But for most applications these days, you will need this packaged into an OCI compatible (i.e. Docker) image for its ultimate deployment to an orchestrator … Java: Creating Docker image for Spring Boot web app using gradle
If you have enabled Actuator in your Spring Boot application, you can add custom status metrics to the standard health check at ‘/actuator/health’. Additionally, your custom health indicator can signal an UP/DOWN status that propagates to the main level status and can then be used by an external monitoring/alerting solutions or as an indicator to … Java: adding custom health indicator to Spring Boot Actuator
If you have enabled Actuator and the ‘micrometer-registry-prometheus’ dependency in your Spring Boot application, then you will have a new ‘/actuator/prometheus’ web endpoint that returns general information about threads, garbage collection, disk, and memory. This information is delivered in standard prometheus formatting as plaintext, with one metric per line. This is exactly the type of … Java: Adding custom metrics to Spring Boot Micrometer Prometheus endpoint
If you have enabled Actuator in your Spring Boot application, then in addition to the standard endpoints for health and metrics, you can also add your own endpoint to expose custom metrics. In its simplest form, this is done by adding a @Component and @Endpoint annotation to a custom class that returns the metrics, and … Java: exposing a custom Actuator endpoint with Spring Boot
Although kubectl has the native ability to query on labels, it does not provide the same support for annotations. This is unfortunate, because the need to target deployments and services by annotation is commonplace. You can, however, use the power of jsonpath to find these objects. Below is an example of finding all Services that … Kubernetes: query by annotation with kubectl
It is easy to export the full manifest of an object from a Kubernetes cluster, but it will have extraneous accounting fields that not only make it hard to visually assess and compare to its original form, but can also cause a re-apply to fail. Using a combination of the jq and yq utilities, we … Kubernetes: export a clean yaml manifest that can be re-imported
If you are configuring Istio/ASM ingress gateways with a BackendConfig for specifying health checks, timeouts, or Cloud Armor policies, then you need to ensure that your GKE cluster has the HttpLoadBalancing feature enabled. If this feature is not enabled, you will see an error message like below when attempting to apply the BackendConfig manifest: unable … GCP: Enable HttpLoadBalancing feature on Cluster to avoid errors when applying BackEndConfig
Update Sep 2023: tested on Kubeadm 1.28, MetalLB v0.13.9, ingress-nginx v1.8.1 Kubeadm is a tool for quickly bootstrapping a minimal Kubernetes cluster. In this article, I will show you how to deploy a three-node Kubeadm cluster on Ubuntu 22 nodes that are created using Terraform and a local KVM libvirt provider. Ansible is used for … KVM: kubeadm cluster on KVM using Ansible
With Workload Identity enabled on a GKE cluster, your container can access Google Cloud API services (Compute Engine, Storage, etc.) using a Kubernetes Service Account (KSA). This is done by having the container run as the KSA, where the KSA has been bound to the Google Service Account (GSA). This is the recommended way of … GCP: running a container on a GKE cluster using Workload Identity
A Kubernetes Service Account (KSA) can be used to provide least-privileged access to a pod for a cluster that has Role-based access control (RBAC) enabled. This is done by making the KSA the subject in an RBAC role. But it can be challenging to discover and test whether the KSA has the correct set of … Kubernetes: testing RBAC authorization of a Kubernetes Service Account
When configuring networks and loadbalancers, sometimes you need the network CIDR block used by Services of a Kubernetes cluster. There are various ways to pull this information from different Kubernetes implementations, but one trick that works across implementations is looking at the error message from kubectl if you attempt to create a service at an … Kubernetes: retrieving services and pods network CIDR block from cluster
GKE cluster upgrades do not need to be a manual process. GKE clusters can be auto upgraded by subscribing the cluster to an appropriate release channel and assigning a sensible maintenance window. As long as adequate pod disruption budgets, replicas, and ingress are configured, these upgrades can happen without interrupting availability. To check the current … GCP: Enabling autoUpgrade for node-pools to reduce manual maintenance
Anthos GKE on-prem is a managed platform that brings GKE clusters to on-premise datacenters. This product offering brings best practice security measures, tested paths for upgrades, basic monitoring, platform logging, and full enterprise support. Setting up a platform this extensive requires many steps as officially documented here. However, if you want to practice in a … Kubernetes: Anthos GKE on-prem 1.11 on nested VMware environment
Anthos GKE on-prem is a managed platform that brings GKE clusters to on-premise datacenters. In this article, I will be following the steps required to upgrade from Anthos 1.10 to 1.11 on VMware. The instructions provided here are assuming you have used the Ansible scripts and Seed VM described in my previous Anthos 1.10 installation … Kubernetes: major version upgrade of Anthos GKE on-prem from 1.10 to 1.11
Bash scripts will often assume that they are being invoked from the same directory where they are located. The script may use relative paths to configuration files or logs that it expects. The problem is that one cannot assume that the directory a script lives in is necessarily the one it is being invoked from. … Bash: current directory versus directory of script
If you have a Bash script that needs to be sourced (versus directly executed), you can enforce this by checking the execution context in the script. Here is an example of setting a flag that can be evaluated and then acted upon. (return 0 2>/dev/null) && sourced=1 || sourced=0 if [ $sourced -eq 0 ]; … Bash: test whether script is invoked directly or sourced
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads on a production Kubernetes cluster, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a series of previous articles, we … Python: New Relic Agent for Gunicorn app deployed on Kubernetes
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of … Python: New Relic instrumentation for Flask app deployed with Gunicorn
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. Running Flask applications directly is great for development and testing of the basic request/response flow, but you need gunicorn to handle production level loads, concurrency, logging, and timeouts. In this article, I will show you how to build a Docker image … Python: Building an image for a Flask app served from Gunicorn
The ‘gcloud compute instances move‘ command is convenient for moving VM instances from one region to another, but only works within a narrow scope of OS image types and disks. For example, only older non-UEFI OS images can be moved with this command. Trying to move even the simplest Ubuntu bionic/focal or Debian bullseye/buster VM … GCP: Moving a VM instance to a different region using snapshots
Anthos Policy Controller enables enforcement of compliance, security, and organizational policies on GKE clusters. These might be best-practice policies coming from internal Architectural standards, or technical policies used to define/constrain resources, or audit requirements stemming from legal regulation. Anthos Policy Controller is built upon the open-source Open Policy Agent (OPA) Gatekeeper, which uses a Kubernetes … GCP: Enable Policy Controller on a GKE cluster
GitHub has a CLI tool that allows you to perform many of the operations you may currently perform in the web UI. Among its many uses, the ‘gh‘ utility will allow you to fork repositories, modify issues, merge PR, and create releases all from the console for ease of use or automation. Below I will … GitHub: CLI tool for repository operations
Since the announcement of CVE-2022-24765, newer git clients from the Ubuntu security and archive package repositories may throw errors about “unsafe repository … is owned by someone else” if directories are not owned by your personal user id. First, try to resolve the issue by running the command suggested in the error message. # attempt … Ubuntu: install latest git client from PPA to fix ‘unsafe repository’ errors
Anthos Config Management (ACM) brings the power of GitOps to your GKE clusters. Instead of needing to manually keep deployments current on a cluster or group of clusters, you can push changes to a git repository and the Config Sync component will periodically poll and attempt to reach the new state described by your git … GCP: Enable Anthos Config Management (ACM) on a GKE cluster
kustomize is typically used to overlay a base set of yaml, but it also has the ability to leverage existing Helm charts, and overlay a set of custom values with HelmChartInflationGenerator. In this article, I will use kustomize to deploy the Bitnami NGINX Helm chart with overridden values that provide a customized nginx.conf and custom … Kubernetes: kustomize with Helm charts
The power of kustomize lies in its ability to transform yaml, and one of the methods for this is patchesStrategicMerge. Where the strategic merge patch excels is in inserting elements and replacing values, allowing you to specify the desired patch using the same indentation level as the target, which makes the intended result very intuitive. … Kubernetes: kustomize transformations with patchesStrategicMerge