Kubernetes: kustomize overlay to enrich a base resource

With kustomize built into the kubectl CLI since version 1.14, there is little reason not to take advantage of this overlay system to deploy components to your Kubernetes cluster.

Kustomize has the advantage that it is purpose built to understand and validate yaml and Kubernetes CRD, as opposed to bespoke templating solutions using sed/envsubst, Ansible, or yq.  And Helm is a great tool, but is heavier and comes with requirements for packaging and building.

In this article, we will build a base NGINX deployment and then enrich the definition using an overlay that can dynamically restart when its backing configmap changes.

Prerequisites

The only environmental prerequisites are that you have kubectl version 1.14+ (which has kustomize built into the CLI commands), and that your KUBECONFIG environment variable is pointed at a Kubernetes config.

Then download my project from github for use in the sections below.

kubectl version --short 

# grab project from github
sudo apt install git -y
git clone https://github.com/fabianlee/kustomize-overlays-with-reload.git
cd kustomize-overlays-with-reload

root_dir=$(realpath .)

kustomize base

The first thing we will do is examine the base resource for ‘nginx-index-html’.  This is a minimalist definition for an NGINX container that returns an index.html that is sourced from a configmap.

cd $root_dir/base/nginx-index-html

$ cat kustomization.yaml

namespace: nginx-index

resources:
- namespace.yaml
- deployment.yaml
- service.yaml

configMapGenerator:
- name: nginx-cm
  namespace: default
  files:
  - cm-index.html

generatorOptions:
  disableNameSuffixHash: true

The resource components create a namespace “nginx-index”, then place the deployment and service inside this namespace.  The content delivered from NGINX comes from the configmap based on the local file “cm-index.html”.

<html>
<head></head>
<body>Hello, World!</body>
</html>

The deployment.yaml defines the volume and volumeMount that place this configmap at /usr/share/nginx/html/index.html and therefore delivered whenever a GET HTTP request is invoked.

Deploying this to your Kubernetes cluster is as simple as using the normal apply command but adding the “-k” flag to indicate the use of kustomize.

# view the yaml that will be created
kubectl kustomize

# now deploy yaml
kubectl apply -k .

# test deployment using curl located inside container
kubectl exec -it deployment/nginx-deployment -n nginx-index -- curl http://localhost

<html>
<head></head>
<body>Hello, World!</body>
</html>

kustomize overlay

To realize the power of kustomize, we will now take that same base NGINX resource, but now enrich it using an overlay.

Let’s use stakater’s Reload operator to watch for changes in the index.html configmap, and automatically do a rolling restart if we modify the contents.  The Reload operator can be deployed like below.

cd $root_dir/base/reloader
kubectl apply -k .

To use this Reload operator, we need to add an annotation to the deployment indicating that changes in its configmap should trigger a rolling restart. Instead of modifying the base, we will use an overlay to add this feature.

cd $root_dir/overlays/nginx-index-with-reloader

cat kustomization.yaml

resources:
- ../../base/nginx-index-html

namespace: nginx-reloader
nameSuffix: "-reloader"

# add annotation that allows configmap reloading
# 'nginx-cm' is the name of configmap, BUT with nameSuffix above, need to tack on '-reloader'
commonAnnotations:
  configmap.reloader.stakater.com/reload: nginx-cm-reloader

configMapGenerator:
- name: nginx-cm
  behavior: replace
  files:
  - cm-index.html

generatorOptions:
  disableNameSuffixHash: true

Notice that we refer back to the base definition, “../../base/nginx-index-html” to pull our original definition but in a unique namespace of “nginx-reloader”.  Then we redefine the configmap with a replace behavior so we can use our own version of “cm-index.html”.

And to enroll in the Reload operator’s logic, we add the “configmap.reloader.stakater.com/reload” annotation to the overlay definition.

To test this new overlay, use the commands below.

# view new overlay definition
kubectl kustomize

# apply overlay definition
kubectl apply -k .

# test using internal curl
kubectl exec -it deployment/nginx-deployment-reloader -n nginx-reloader -- curl http://localhost

<html><head></head>
<body>
<h1>nginx-index-html enhanced with reloader</h1>
</body>
</html>

You can see that the index.html content now comes from the overlay’s cm-index.html.

But we also need to test the auto redeployment if the configmap changes.

# change the overlay contents
sed -i 's/reloader/reloader-updated1/' cm-index.html

# reapply the overlay
# the Reload operator will sense the configmap change
kubectl apply -k .

# test using internal curl, notice the content changed
kubectl exec -it deployment/nginx-deployment-reloader -n nginx-reloader -- curl http://localhost

<html><head></head>
<body>
<h1>nginx-index-html enhanced with reloader-updated1</h1>
</body>
</html>

This is just one example of how overlays can be used to enrich and transform base definitions.   The addition, removal, and replacement of entire sections can be performed using the various kustomize patch types.

 

REFERENCES

kubernetes.io, kustomize

kubectl, three inline patch types (patchesStrategicMerge,patchesJSON6902,patches)

RFC 6902

Arun Kumar Singh, kustomize intro