Anthos GKE on-prem is a managed platform that brings GKE clusters to on-premise datacenters. In this article, I will be following the steps required to upgrade from Anthos 1.10 to 1.11 on VMware. The instructions provided here are assuming you have used the Ansible scripts and Seed VM described in my previous Anthos 1.10 installation … Kubernetes: major version upgrade of Anthos GKE on-prem from 1.10 to 1.11
Bash scripts will often assume that they are being invoked from the same directory where they are located. The script may use relative paths to configuration files or logs that it expects. The problem is that one cannot assume that the directory a script lives in is necessarily the one it is being invoked from. … Bash: current directory versus directory of script
If you have a Bash script that needs to be sourced (versus directly executed), you can enforce this by checking the execution context in the script. Here is an example of setting a flag that can be evaluated and then acted upon. (return 0 2>/dev/null) && sourced=1 || sourced=0 if [ $sourced -eq 0 ]; … Bash: test whether script is invoked directly or sourced
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads on a production Kubernetes cluster, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a series of previous articles, we … Python: New Relic Agent for Gunicorn app deployed on Kubernetes
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of … Python: New Relic instrumentation for Flask app deployed with Gunicorn
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. Running Flask applications directly is great for development and testing of the basic request/response flow, but you need gunicorn to handle production level loads, concurrency, logging, and timeouts. In this article, I will show you how to build a Docker image … Python: Building an image for a Flask app served from Gunicorn
The ‘gcloud compute instances move‘ command is convenient for moving VM instances from one region to another, but only works within a narrow scope of OS image types and disks. For example, only older non-UEFI OS images can be moved with this command. Trying to move even the simplest Ubuntu bionic/focal or Debian bullseye/buster VM … GCP: Moving a VM instance to a different region using snapshots
Anthos Policy Controller enables enforcement of compliance, security, and organizational policies on GKE clusters. These might be best-practice policies coming from internal Architectural standards, or technical policies used to define/constrain resources, or audit requirements stemming from legal regulation. Anthos Policy Controller is built upon the open-source Open Policy Agent (OPA) Gatekeeper, which uses a Kubernetes … GCP: Enable Policy Controller on a GKE cluster
GitHub has a CLI tool that allows you to perform many of the operations you may currently perform in the web UI. Among its many uses, the ‘gh‘ utility will allow you to fork repositories, modify issues, merge PR, and create releases all from the console for ease of use or automation. Below I will … GitHub: CLI tool for repository operations
Since the announcement of CVE-2022-24765, newer git clients from the Ubuntu security and archive package repositories may throw errors about “unsafe repository … is owned by someone else” if directories are not owned by your personal user id. First, try to resolve the issue by running the command suggested in the error message. # attempt … Ubuntu: install latest git client from PPA to fix ‘unsafe repository’ errors
Anthos Config Management (ACM) brings the power of GitOps to your GKE clusters. Instead of needing to manually keep deployments current on a cluster or group of clusters, you can push changes to a git repository and the Config Sync component will periodically poll and attempt to reach the new state described by your git … GCP: Enable Anthos Config Management (ACM) on a GKE cluster
kustomize is typically used to overlay a base set of yaml, but it also has the ability to leverage existing Helm charts, and overlay a set of custom values with HelmChartInflationGenerator. In this article, I will use kustomize to deploy the Bitnami NGINX Helm chart with overridden values that provide a customized nginx.conf and custom … Kubernetes: kustomize with Helm charts
The power of kustomize lies in its ability to transform yaml, and one of the methods for this is patchesStrategicMerge. Where the strategic merge patch excels is in inserting elements and replacing values, allowing you to specify the desired patch using the same indentation level as the target, which makes the intended result very intuitive. … Kubernetes: kustomize transformations with patchesStrategicMerge
The power of kustomize lies in its ability to transform yaml, and to that end it has built-in support for the JSON 6902 RFC specification. This RFC specifies the syntax for adding, removing, replacing, and moving elements and values in a yaml file. And since Kubernetes manifests are yaml files, it makes perfect sense to … Kubernetes: kustomize transformations with patchesJson6902
Kubernetes has a rich way of expressing volumes/ volumeMounts for mounting files, emptyDir for ephemeral directories, and env/envFrom for adding environment variables to your container definition running on a Kubernetes cluster. However, if you are actively iterating on the development of an image, it may slow you down to require a deployment to a remote … Kubernetes: volumeMount, emptyDir, and env equivalents during local Docker development
With kustomize built into the kubectl CLI since version 1.14, there is little reason not to take advantage of this overlay system to deploy components to your Kubernetes cluster. Kustomize has the advantage that it is purpose built to understand and validate yaml and Kubernetes CRD, as opposed to bespoke templating solutions using sed/envsubst, Ansible, … Kubernetes: kustomize overlay to enrich a base resource
At some point you may need to schedule a maintenance window for your solution But that doesn’t mean the end-user traffic or client integrations will stop requesting the services from the GCP external HTTPS LB that fronts all client requests. The VM instances and GKE clusters that normally respond to requests may not be able … GCP: Cloud Function to handle requests to HTTPS LB during maintenance
GCP Cloud Functions have taken a step forward with the 2nd generation release. One of the biggest architectural differences is that now multiple request can run concurrently on a single instance, enabling large traffic loads. In this article, I will show you how to deploy a simple Python Flask web server as a 2nd gen … GCP: Deploying a 2nd gen Python Cloud Function and exposing from an HTTPS LB
The Compute Engine default service account is automatically generated for your project with the Editor role, and by default is attached to all VM instances created in the project. You can pull the exact id using gcloud. gcloud iam service-accounts list –filter=”displayName:’Compute Engine default service account'” –format=’value(email)’ The syntax will be ${project_id}-compute@developer.gserviceaccount.com. If you want … GCP: VM instances running as the Compute Engine default service account
If you have unmanaged GCP VM instances running services on insecure ports (e.g. Apache HTTP on port 80), one way to secure the public external traffic is to create an external GCP HTTPS load balancer. Conceptually, we want to expose a secure front to otherwise insecure services. While the preferred method would be to secure … GCP: global external HTTPS LB for securely exposing insecure VM services
If you have unmanaged GCP VM instances running services on insecure ports (e.g. Apache HTTP on port 80), one way to secure the internal communication coming from other internal pods/apps is to create an internal GCP HTTPS load balancer. Conceptually, we want to expose a secure front to otherwise insecure services. While the preferred method … GCP: internal HTTPS LB for securely exposing insecure VM services
A Bash script can often run into the situation where a utility in the pipeline creates a file, but because of an unexpected error, the size of the resulting file is zero bytes. And if that situation is not prepared for, the mere existence of the file might signal success to the following commands in … Bash: test both file existence and size to avoid signalling success
No matter how highly available your services, there may still be significant backend events that require planned maintenance. During this downtime, you should still reply to end users and service integrations with a proper response. In this article, I will show you how to configure your GCP HTTPS Loadbalancer so that a single maintenance service … GCP: serving a maintenance page using an HTTPS LB and container native routing
If you need to delete a GKE instance from a node pool, you cannot simply treat the node as a raw VM instance. You must delete the VM instance from the managed instance group of which it is a member. A cluster will have an instance group for each region. In a zonal cluster, the … Kubernetes: deleting a GKE node from a managed instance node pool
If your intent is to delete all the objects in a namespace, but the command is not completing, emptying the namespace finalizer will often allow the deletion to finish. For example, if you have tried deleting the “my-namespace” like below and it will not complete. kubectl delete ns my-namespace –force –grace-period=0 Then as written by … Kubernetes: emptying the finalizers for a namespace that will not delete
If you are using GCP HTTPS LB to deliver your public services, be sure to apply an explicit SSL Policy that controls how TLS is negotiated with clients. Setting a SSL policy allows you to control minimum version of TLS as well as available cipher families. A basic SSL policy that limits clients to … GCP: enabling SSL policies on HTTPS LB Ingress
It is not necessary to create an independent GCP HTTPS LB or other improvisation to redirect insecure HTTP traffic to your HTTPS load balancer. The existing public Ingress can reference a FrontendConfig object that specifies redirection to HTTPS. Below is a FrontendConfig definition that can redirect the insecure traffic. apiVersion: networking.gke.io/v1beta1 kind: FrontendConfig metadata: name: … GCP: HTTP to HTTPS redirection using HTTPS LB Ingress
GKE Autopilot reduces the operational costs of managing GKE clusters by freeing you from node level maintenance, instead focusing just on pod workloads. Costs are accrued based on pod resource consumption and not on node resource sizes or node count, which are managed by Google. Since you no longer own the node level, there are … GCP: Private GKE cluster in Autopilot mode using Terraform
As opposed to public GKE clusters which have their IP addresses exposed, private GKE clusters use private internal IP addresses. This provides an enhanced security stance, but also means we need a solution such as Anthos Service Mesh to explicitly expose our services. In our previous article, we built a private GKE cluster using Terraform. … GCP: Private GKE Cluster with Anthos Service Mesh exposing services
As opposed to public GKE clusters which have their IP addresses exposed, private GKE clusters use private internal IP addresses that offer a level of security and segmentation that should always be preferred. In this article, I will show you how to create a private GKE cluster with Terraform. The endpoint is private also for … GCP: Private GKE Cluster with private endpoint using Terraform
If you are using Anthos Service Mesh to deliver your public applications from a GFE HTTPS LB, I would strongly suggest enabling Cloud Armor which is a WAF (web application firewall) that can mitigate and defend against a variety of attacks such as cross-site scripting and denial of service. As a summary overview, the first … GCP: enabling Cloud Armor on GCP HTTPS LB for Anthos Service Mesh
For bulk administration of hosts via ssh, there are powerful tools such as Ansible for the job. However, if you need to automate ssh for a one-off task, you may find the remote host systems throwing up interactive authentication prompts for sudo that need to be answered. If you need to run sudo commands that … Bash: automating ssh login and sudo that require interactive login
Killing a defunct zombie process means identifying and killing the parent process. In this article, I will show how you can force a zombie process so that you can practice removing it. A simple way of forcing a zombie process is to start a process that has a short sleep in the background, immediately invoking … Bash: identifying and killing a zombie child processes
Handling signals from within Bash has a simple syntax and can be used to handle user interruption with Ctrl-C (SIGINT), as a cleanup hook at the end of processing (SIGEXIT), or even as a custom messaging mechanism (SIGUSR1). Trapping a signal is as easy as specifying a custom function name and the signal name as … Bash: trapping signals during script processing
When calling processes within a script, there may be situations where you need to put a time limit on the call due to significant blocking times. Perhaps a DNS or network call would block if certain firewall rules do not exist yet, or a long-running API call would need to be considered in error if … Bash: using timeout to put time limit on invoked commands
If the logic in your Bash script needs to force the last exit code variable “$?”, you can accomplish that with an exit inside a subprocess. For example, here is a script snippet. $(exit 99) echo “last process exit code was $?” This would return the output below. last process exit code was 99 This … Bash: forcing the process exit code using a subshell
If you delete the entire nginx namespace and reinstall again via helm chart, your nginx admission controller may throw a “x509 certificate signed by unknown authority” message when you attempt to create an nginx ingress. This will happen regardless if the ingress is using http only or secure https. And also whether or not the … Kubernetes: ingress-nginx-controller-admission error, x509 certificate signed by unknown authority
Instead of deploying a pod or service and periodically checking its status for readiness, or having your automation scripts wait for a certain number of seconds before moving to the next operation, it is much cleaner to use ‘kubectl wait’ to sense completion. Wait for pod Here is how you would wait for READY status … Kubernetes: using kubectl to wait for condition of pods, deployments, services
The ‘export’ flag in kubectl was deprecated in 1.18, but you still need a way to dump the yaml of Kubernetes objects, often for the purpose of reapplying changes. The problem with doing a simple ‘get’ using a yaml output like below is that additional internal accounting attributes like ‘selfLink’, ‘creationTimestamp’, ‘uid’, will be present … Kubernetes: alternative to export for removing internal fields from yaml
The kubectl jsonpath has a built-in ‘range’ directive that allows you to iterate lists, and then extract elements from each list item. For example, if you wanted a list of all the pod names and their namespace in CSV format, you could use the following: # pods and their namespace with just kubectl kubectl get … Kubernetes: jsonpath range to iterate list and extract fields