Kubernetes: deploying Kyverno for cluster policy control

Kyverno is an open-source project that manages and enforces policies within a Kubernetes cluster.  The policy definitions are defined as yaml and deployed as Kubernetes objects.

Kyverno has become popular for its Kubernetes-specific policy engine and declarative rule definitions (as opposed to a general policy engine like OPA/Gatekeeper that use a domain specific language).  It also has rich support for CLI tools that allow evaluation of policies from within pipelines.

Deploy Kyverno into Kubernetes cluster

Per the official docs, use Helm to deploy a standalone instance into your Kubernetes cluster (non-HA).

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
# view latest chart versions available
helm search repo kyverno/kyverno --versions | head -n10

# fresh install
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

# show all objects created
kubectl get all -n kyverno
# show webhooks created
kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration -A | grep kyverno
kubectl get Mutatingwebhookconfigurations -A | grep kyverno
# show webhook excluded namespaces (kube-system,kyverno)
kubectl get cm -n kyverno kyverno -o=jsonpath="{.data.webhooks}" | jq .

# ensure all deployments are ready before moving forward
kubectl rollout status deployment -n kyverno

kyverno CLI tool installation on Debian/Ubuntu

Below are the steps for manual installation of the CLI tool, although krew could be used as well.

latest_version=$(curl -sL https://api.github.com/repos/kyverno/kyverno/releases/latest | jq -r ".tag_name")
wget https://github.com/kyverno/kyverno/releases/download/$latest_version/kyverno-cli_${latest_version}_linux_x86_64.tar.gz

# extract and place on PATH
tar xvfz kyverno-cli_${latest_version}_linux_x86_64.tar.gz
sudo cp kyverno /usr/local/bin/.

# validate
kyverno version

Kyverno policy writing

There are 50+ policy examples on the official site, and this is the best place to understand the scope of what is possible with Kyverno policies.

Although security may be the first word that comes to mind when hearing “policy”, the Kyverno mutation engine can enrich deployment manifest in ways that simplify the life of developers (or Helm chart authors).

The “Writing Policies” page of the official documentation is where you can get into the details of the declarative rules, but for right now, let’s just look at two simple examples in the following sections.

Kyverno policy test #1: namespace policy

We will start with a policy on namespaces that requires a label named “owner” and a label named “env” that can be one of dev|staging|prod (rule originally authored by Glen Yu).

# get policy and apply
wget https://raw.githubusercontent.com/fabianlee/blogcode/refs/heads/master/k8s/kyverno/kyv-require-ns-label.yaml
kyverno apply kyv-require-ns-label.yaml --cluster --v=5

# EXPECTED kyverno policy control will stop this creation
kubectl create ns this-ns-test-will-fail

# logs will show policy name that blocked
kubectl logs deployment/kyverno-admission-controller -n kyverno | grep "blocking "

# kyverno policy allows because 'env' and 'owner' labels assigned
kubectl apply -f - << EOF
apiVersion: v1
kind: Namespace
metadata:
  name: kyvtest
  labels:
    env: dev
    owner: me
EOF

# logs will show policy allowed
kubectl logs deployment/kyverno-admission-controller -n kyverno | grep " passed"

Kyverno policy test #2: pod resource mutation

The second policy is a slight variation on the official “Add Default Resources“, that adds memory/cpu resource limits to pods that do not define them.

In our case, we have added an exclusion for the “default” namespace, which means it will NOT act upon pods in the default namespace.  But it should still mutate pods created in other namespaces, such as the “kyvtest” namespace just created in the section above.

# get policy and apply
wget https://raw.githubusercontent.com/fabianlee/blogcode/refs/heads/master/k8s/kyverno/add-default-resources.yaml
kubectl apply -f add-default-resources.yaml

# get simple deployment manifest that does NOT define any resource limits
wget https://raw.githubusercontent.com/fabianlee/blogcode/refs/heads/master/k8s/tiny-tools.yaml

# deploy to 'default' namespace and newly created 'kyvtest' namespace
kubectl apply -f tiny-tools.yaml -n default
kubectl apply -f tiny-tools.yaml -n kyvtest

# wait for deployment
kubectl rollout status deployment tiny-tools -n kyvtest
kubectl rollout status deployment tiny-tools -n default
# check resource values: will be empty for 'default' ns, and set to cpu=100m/memory=100Mi for kyvtest
kubectl get pods -l app=tiny-tools -A -o=jsonpath="{range .items[*]}{.metadata.namespace},{.metadata.name},{.spec.containers[*].resources}{'\n'}{end}"

 

REFERENCES

Kyverno site

Kyverno github source

Kyverno sample policies

Kyverno policy that adds explicit resource defaults to pods

Kyverno helm chart values.yaml

Kyverno, writing policy rules

OPA/Gatekeeper overview

nirmata.com, 5 reasons Kyverno is better than OPA

nirmata.com, deeper analysis of Kyverno versus OPA

Amine Raje, Kyverno versus OPA

Glen Yu, why I prefer Kyverno over OPA

Open Policy Agent OPA example

atlassian, upgrading a helm chart

 

NOTES

remove kyverno from cluster

helm uninstall kyverno -n kyverno

Exclude namespaces from evaluation at global level using resourceFilters

# show current namespaces excluded from kyverno policy enforcement
kubectl get cm -n kyverno kyverno -o=jsonpath="{.data.resourceFilters}" | sed 's/ /\n/g' | grep '^\[\*/\*,'

# append additional filters that makes 'default' namespace ignored
helm upgrade --install kyverno kyverno/kyverno -n kyverno --create-namespace --set "config.resourceFiltersIncludeNamespaces={default,istio-system}"
helm get values kyverno -n kyverno -n kyverno
# resource filters will now include 'default' namespace filter
kubectl get cm -n kyverno kyverno -o=jsonpath="{.data.resourceFilters}" | sed 's/ /\n/g' | grep '^\[\*/\*,'

kyverno upgrade using helm

# view latest chart versions available
helm search repo kyverno/kyverno --versions | head -n10
# upgrade to newer chart
helm upgrade --install kyverno kyverno/kyverno -n kyverno --create-namespace --version=3.3.3