Vault: synchronizing secrets from Vault to Kubernetes using Vault Secrets Operator


Categories: Containers, DevOps, Kubernetes

The Vault Secrets Operator is a Vault integration that runs inside a Kubernetes cluster and synchronizes Vault-level secrets to Kubernetes-level secrets.

This secret synchronization happens transparently to the running workloads, without any need to retrofit existing images or manifests.

In this article, I will show how to:

  • Install the Vault Secrets Operator (VSO)
  • Configure the VSO to authenticate to an external Vault server using JWT auth mode
  • Create Vault-level secret for key/value pairs, synchronize to Kubernetes-level secret
  • Deploy a running container that does a rolling restart whenever Vault-level secrets are made
  • Create a Vault-level secret for a TLS cert+key, synchronize to Kubernetes-level TLS secret

Prerequisites

Local utilities

In order to parse json results, generate certificates, and analyze the JWT used in this article, install the utilities below.

# json parser
sudo apt install -y jq

# step utility for certificate generation
wget -q https://github.com/smallstep/cli/releases/download/v0.25.0/step_linux_0.25.0_amd64.tar.gz
tar xvfz step_linux_0.25.0_amd64.tar.gz step_0.25.0/bin/step --strip-components 2
rm step*.gz

# jwker for JWK/PEM conversions
wget -q https://github.com/jphastings/jwker/releases/download/v0.2.1/jwker_Linux_x86_64.tar.gz
tar xvfz jwker_Linux_x86_64.tar.gz jwker
rm jwker*.gz

Helm

You can read my other article on installing Helm on Ubuntu, or read the official Helm docs.

Kubernetes cluster

Feel free to use any Kubernetes implementation (k3s, minikube, kubeadm, GCE, EKS, etc).

If you want to create a fresh Kubernetes cluster just for this article, minikube is used extensively in the Vault tutorials.  If you do not already have minkube installed, see the official starting documentation for installation instructions.  There are other installation references available if you need further detail (1, 2, 3).

External Vault Server

We want to emulate a Vault Server external to the cluster, if you do not already have one to work with, starting a Vault sever up in development mode on your local host is an easy way to test.

The official Vault installation document is here, below are instructions for an Ubuntu host as detailed here.

# download trusted key
sudo apt update && sudo apt install gpg
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
sudo chmod go+r /usr/share/keyrings/hashicorp-archive-keyring.gpg

# add to repo list
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

# install Vault
sudo apt update && sudo apt install -y vault

# start Vault, use IP address of your host server
vaultIP=<yourLocalIPAddress>
echo "Starting vault on ${vaultIP}:8200, this IP needs to be used from cluster later !"
vault server -dev -dev-root-token-id root -dev-listen-address $vaultIP:8200 -log-level debug

Copy down the $vaultIP value because it will need to be used later.

Install the Vault Secrets Operator using Helm

Install the Vault Secrets Operator (VSO) per its Helm chart settings (values.yaml).

# add local helm repo
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update hashicorp
helm search repo hashicorp/vault-secrets-operator

# custom namespace for VSO
vso_ns=vault-secrets-operator

# install VSO
helm upgrade --install vault-secrets-operator hashicorp/vault-secrets-operator --namespace $vso_ns --create-namespace

# show created objects
$ kubectl get all -n $vso_ns
NAME                                                             READY   STATUS    RESTARTS   AGE
pod/vault-secrets-operator-controller-manager-5f9dcdb878-4fbh9   2/2     Running   0          37s

NAME                                             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/vault-secrets-operator-metrics-service   ClusterIP   10.109.2.23           8443/TCP   37s

NAME                                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-secrets-operator-controller-manager   1/1     1            1           37s

NAME                                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-secrets-operator-controller-manager-5f9dcdb878   1         1         1       37s

We only installed the minimal VSO Operator, and will configure it in later sections.

Create Vault secret

Create a Vault secret and policy that VSO will later synchronize into a Kubernetes secret.

# external Vault server address
vaultIP=<yourVaultIPAddress>
export VAULT_ADDR=http://$vaultIP:8200

# provide proper credentials for Vault access (using dev mode here)
vault login root

# create Vault secret for hello app ("Hello, $greeting")
vault kv put secret/webapp/hello greeting=Galaxy

# create Vault policy that can see secrets
# 'data' path needs to be inserted when dealing with API level
echo 'path "secret/data/webapp/*" { capabilities=["read"] }' | vault policy write vso-policy -

Configure Vault JWT auth and secret

We must enable JWT auth mode between the external Vault server and VSO installed on the cluster.

Fetch Cluster OIDC public key

We need the Cluster (issuer) OIDC public key in order to verify the JWT source and integrity.  This can be retrieved from the ‘/openid/v1/jwks’ endpoint of the cluster.

# query the jwks_uri at private endpoint to get public key
kubectl get --raw /openid/v1/jwks | jq .keys[0] | tee cluster.jwk

# convert jwk OIDC public key to pem format for use by Vault
./jwker cluster.jwk > cluster.pem

Grabbing the cluster OIDC public key manually as shown (versus self-discovery) helps avoids multiple issues with ensuring the endpoint is network-available, has a public ingress, and allows anonymous access.

Create Kubernetes service account

Create a Kubernetes service account that can generate short-lived JWT for authentication.

echo "Creating a service account in the 'default' namespace"
kubectl create sa vso-auth -n default

Configure Vault server for JWT

Enable JWT auth mode for this cluster.  We supply the cluster OIDC public key retrieved earlier, which is required in order to test JWT validity/integrity.

# enable JWT authentication
vault auth enable -path=vsojwt jwt
vault auth list

# create authentication endpoint for this cluster using public key of cluster
# not using self-discovery because too many connectivity, ingress, and auth issues can exist
vault write auth/vsojwt/config jwt_validation_pubkeys=@cluster.pem

Then create the Vault role tied to the service account ‘default:vso-auth’ and using the audience “audience-vso-auth”.

# create Vault role associated to service account
vault write auth/vsojwt/role/vso role_type=jwt ttl=10m token_policies=vso-policy bound_audiences=audience-vso-auth bound_subject=system:serviceaccount:default:vso-auth user_claim=sub verbose_oidc_logging=true

Configure the Vault Secrets Operator

We can now finish off the VSO configuration on the cluster which tells it how to connect back to Vault.  First, create a VaultConnection object.

# create VaultConnection CRD
wget https://raw.githubusercontent.com/fabianlee/blogcode/master/vault/vso/vaultconnection.yaml
cat vaultconnection.yaml | vso_ns=$vso_ns vault_url=http://192.168.2.239:8200 envsubst | kubectl apply -f -

# should see successful 'VaultConnection accepted' event
kubectl logs deployment/vault-secrets-operator-controller-manager -n $vso_ns | grep VaultConnection

Then create the VaultAuth object which ties the VaultConnection to the JWT auth mode settings (role, service account, audience, etc).

# create VaultAuth CRD
wget https://github.com/fabianlee/blogcode/raw/master/vault/vso/vaultauth-jwt.yaml
cat vaultauth-jwt.yaml | vso_ns=$vso_ns envsubst | kubectl apply -f -

# should see successful 'Successfully handled VaultAuth' event
kubectl logs deployment/vault-secrets-operator-controller-manager -n $vso_ns | grep vso-staticsecret

Then create the VaultStaticSecret which ties a Vault secret to a Kubernetes secret, and specifies any workloads that need to be restarted if the secret changes.

# create VaultStaticSecret CRT
wget https://github.com/fabianlee/blogcode/raw/master/vault/vso/vaultstaticsecret-hello.yaml
cat vaultstaticsecret-hello.yaml | envsubst | kubectl apply -f -

# should see successful 'events Secret synced'
kubectl logs deployment/vault-secrets-operator-controller-manager -n $vso_ns | grep vso-staticsecret-hello | grep -v 'not required'

If everything has been successful, then VSO should have created the Kubernetes-level secret we requested.

# kubernetes secret should now exist
kubectl get secret hello-secret

# kubernetes secret can be seen normally
kubectl get secret hello-secret -o=jsonpath="{.data._raw}" | base64 -d | jq .data
{
  "greeting": "Galaxy"
}

# change vault level secret and wait 15 seconds
vault kv put secret/webapp/hello greeting=Universe
sleep 15

# should see 'events Secret synced' with reason listed as rotation
kubectl logs deployment/vault-secrets-operator-controller-manager -n $vso_ns | grep vso-staticsecret-hello | grep SecretRotated

# change propagated to kubernetes level secret
$ kubectl get secret hello-secret -o=jsonpath="{.data._raw}" | base64 -d | jq .data
{ 
  "greeting": "Universe"
}

Workload restart based on Vault change

VSO also has the ability to do a rolling restart of workloads (Deployment, Daemonset, StatefulSet) when a secret changes.

This is done by adding a rolloutRestartTargets section to the VaultStaticSecret that explicitly lists the workloads to restart when the secret changes.  Below is a snippet from vaultstaticsecret-hello.yaml.

...
spec:
  ...
  rolloutRestartTargets:
    - kind: Deployment
      name: web-hello

Below is the ‘web-hello’ deployment that will have a rolling restart invoked by VSO when we change the Vault secret.

# deploy app that uses value in 'hello-secret'
wget https://github.com/fabianlee/blogcode/raw/master/vault/vso/web-hello.yaml
kubectl apply -f web-hello.yaml

# secret value overrides default and prints out 'Hello, Universe'
$ kubectl exec -it deployment/web-hello -n default -- wget -q http://localhost:8080 -O-
Hello, Universe
request 0 GET /
Host: localhost:8080

# change vault level secret and wait 15 seconds
vault kv put secret/webapp/hello greeting=Neighborhood
sleep 15 

# make sure deployment is ready again
kubectl rollout status deployment web-hello -n default --timeout=30s

# new secret now used 'Hello, Neighborhood'
$ kubectl exec -it deployment/web-hello -n default -- wget -q http://localhost:8080 -O-
Hello, Neighborhood
request 0 GET /
Host: localhost:8080

Testing TLS certificate synchronization

In the previous section we synchronized a secret containing key/value pairs, but VSO synchronization also works with TLS secrets, the same as one might reference from ingress-nginx or Istio.

Create TLS certificate

We will us a self-signed certificate for simplicity of this example.

# example domain for TLS certificate
domain=hello-world.info

# create self-signed cert
./step certificate create $domain $domain.crt $domain.key --profile self-signed --subtle --no-password --insecure

# show self-signed cert
./step certificate inspect $domain.crt --short

Create Vault secret from TLS cert + key

vault kv put secret/webapp/cert tls\.crt=@$domain.crt tls\.key=@$domain.key

Associate Vault Secret to Kubernetes secret

Create the VaultStaticSecret which ties a Vault secret to a Kubernetes secret.

wget https://github.com/fabianlee/blogcode/raw/master/vault/vso/vaultstaticsecret-cert.yaml
kubectl apply -f vaultstaticsecret-cert.yaml

# should see successful 'events Secret synced'
kubectl logs deployment/vault-secrets-operator-controller-manager -n $vso_ns | grep vso-staticsecret-cert

Validate TLS certificate synchronization to Kubernetes

# kubernetes secret of type 'kubernetes.io/tls' should now exist
$ kubectl get secret cert-secret
NAME TYPE DATA AGE
cert-secret kubernetes.io/tls 3 4m24s

# cert loaded into k8s tls secret should match the one in local cert file
diff hello-world.info.crt <(kubectl get secret cert-secret -o=jsonpath="{.data.tls\.crt}" | base64 -d)

# certificate can be inspected with openssl
kubectl get secret cert-secret -o=jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -text -noout

Test certificate refresh

# create new self-signed cert with 72 hour expiration
./step certificate create $domain $domain.crt $domain.key -f --not-after 72h --profile self-signed --subtle --no-password --insecure

# serial number and valid 'to' should have now changed
./step certificate inspect $domain.crt --short

# overwrite Vault secret with this new cert
vault kv put secret/webapp/cert tls\.crt=@$domain.crt tls\.key=@$domain.key

# should see 'events Secret synced' with reason listed as rotation
kubectl logs deployment/vault-secrets-operator-controller-manager -n $vso_ns | grep vso-staticsecret-cert | grep SecretRotated

# should see updated cert with 72 hour 'Not After' validity
kubectl get secret cert-secret -o=jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -text -noout

If you wanted to do a restart of a deployment (e.g. ingress-nginx-controller) after this refresh, add a rolloutRestartTargets section to the VaultStaticSecret, similar to how we did for vaultstaticsecret-hello.yaml.

 

REFERENCES

github fabianlee, code for this article

HashiCorp, installing the Secrets Operator

HashiCorp, Vault Secrets Operator helm chart

HashiCorp, VSO auth methods

Baeldung, different methods of accessing secrets using Spring framework including VSO

Github, Vault Secrets Operator

HashiCorp, Kubernetes vault integration via sidecar versus CSI provider

Github VSO, sample CRD

github issue, with vaultAuth needing service account with namespace scope [pull request]

github VSO issues page

 

NOTES

Prove out service account access to Vault secret

Deploy app using ‘vso-auth’ service account

# deploy sample app using same 'vso-auth' service account
cat tiny-tools-template.yaml | ns=default envsubst | kubectl apply -f -

# grab JWT from container running as 'vso-auth'
kubectl exec -it deployment/tiny-tools-jwt -n default -- cat /var/run/secrets/kubernetes.io/serviceaccount/token > vso-auth.jwt

# informally inspect JWT, should show specialized audience and sub
cat vso-auth.jwt | ./step crypto jwt inspect --insecure

# formally validate JWT
cat vso-auth.jwt | ./step crypto jwt verify --key cluster.jwk --iss https://kubernetes.default.svc.cluster.local --aud audience-vso-auth

Fetch Vault secret using JWT

# enter running container
kubectl exec -it deployment/tiny-tools-jwt -n default -- sh

# set to external vault address from previous section
vaultIP=<yourLocalVaultIP>

# inspection of JWT
cat /var/run/secrets/kubernetes.io/serviceaccount/token | step crypto jwt inspect --insecure

# exchange JWT for Vault token
vault_token=$(curl -Ss http://$vaultIP:8200/v1/auth/vsojwt/login --data "{\"jwt\": \"$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\", \"role\": \"vso\"}" | jq -r ".auth.client_token")
echo "traded short-lived JWT for vault token: $vault_token"

# fetch secret using Vault token
$ curl -s -H "X-Vault-Token: $vault_token" http://$vaultIP:8200/v1/secret/data/webapp/hello | jq '.data.data'
{ 
  "greeting": "Galaxy" 
}