GCP: running a container on a GKE cluster using Workload Identity

With Workload Identity enabled on a GKE cluster, your container can access Google Cloud API services (Compute Engine, Storage, etc.) using a Kubernetes Service Account (KSA).

This is done by having the container run as the KSA, where the KSA has been bound to the Google Service Account (GSA).  This is the recommended way of running workloads that require secure access to Google cloud services.

In this article I will show how enable Workload Identity on your GKE cluster and node pool, and then how to run a container image as a KSA that can run gcloud commands because of its binding to a GSA.

Enable Workload Identity

The first step is to enable Workload Identity at the GKE cluster and node pool level.  First, check the current settings of the GKE cluster.

# list clusters
export project_id=$(gcloud config get-value project)
gcloud container clusters list --format="csv[no-heading](name,location)"

# set variables
cluster_name=xxxxxx
# use either 'region' or 'zone'
location_flag="--region xxxxx"

# set name of node pool
nodepool_name=$(gcloud container node-pools list --cluster=$cluster_name $location_flag --format="value(name)")

# if empty, then workload identity not set at cluster level
# if workload identity enabled, will return ${project_id}.svc.id.goog
gcloud container clusters describe $cluster_name $location_flag --format="value(workloadIdentityConfig.workloadPool)"

# if empty, then workload identity not set at node pool level
gcloud container node-pools describe $nodepool_name --cluster=$cluster_name $location_flag --format="value(config.workloadMetadataConfig.mode)"

If Workload Identity needs to be enabled, then use the below commands.

# enable at cluster level, can take ~20 minutes
gcloud container clusters update $cluster_name $location_flag --workload-pool=${project_id}.svc.id.goog

# enable at node pool level
# takes time because it does serial rebuild of each node in pool
gcloud container node-pools update $nodepool_name --cluster=$cluster_name $location_flag --workload-metadata=GKE_METADATA

Create Google Service Account (GSA)

Create a Google Service Account “gcloud-user” that has the IAM roles required to access the desired Google services.

GSA=gcloud-user
# create account, allow settling with delay
gcloud iam service-accounts create $GSA --project=$project_id
sleep 15

# get fully qualified email for GSA
# syntax will be ${GSA}@${project_id}.iam.gserviceaccount.com
svcEmail=$(gcloud iam service-accounts list --project=$project_id --filter="name ~ ${GSA}@" --format="value(email)")

# add these roles to GSA
roles="roles/container.clusterViewer roles/container.viewer roles/iam.workloadIdentityUser"
for role in $roles; do
  gcloud projects add-iam-policy-binding $project_id --member=serviceAccount:$svcEmail --role=$role > /dev/null
done

echo "show final '$svcEmail' roles"
gcloud projects get-iam-policy $project_id --flatten="bindings[].members" --filter="bindings.members=serviceAccount:$svcEmail" --format="value(bindings.role)"

Create Kubernetes Service Account (KSA)

Now create the KSA with an annotation that references the GSA.  This can be done by applying the my-wi-ksa.yaml manifest with kubectl.

# my-wi-ksa.yaml
---
# binds k8s service account to role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ksa-wi-reader-binding
subjects:
- kind: ServiceAccount
  name: ${KSA}
  namespace: default
roleRef:
  kind: ClusterRole
  name: ksa-wi-reader
  apiGroup: rbac.authorization.k8s.io
---
# role that defines permissions
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ksa-wi-reader
rules:
- apiGroups: [""]
  resources: ["namespaces","nodes", "pods", "secrets", "services", "configmaps","crontabs","persistentvolumes"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["daemonsets", "replicasets", "statefulsets"] # intentionally left out 'deployments'
  verbs: ["get", "list", "watch"]
---
# k8s service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${KSA}
  namespace: default
  annotations:
    iam.gke.io/gcp-service-account: gcloud-user@${project_id}.iam.gserviceaccount.com

Then apply to the cluster.

# envsubst used to replace with env values
export KSA=my-wi-ksa
envsubst < my-wi-ksa.yaml | kubectl apply -f -

Bind KSA and GSA

With the GSA and KSA now created, we can bind them together so that the KSA can impersonate the GSA roles.

# bind GSA and KSA
gcloud iam service-accounts add-iam-policy-binding $svcEmail --role roles/iam.workloadIdentityUser --member serviceAccount:${project_id}.svc.id.goog[default/$KSA]

Create Container running as KSA

The final step is to deploy a container running as the KSA.  From inside this container, the Google Cloud API will be available with the level of permissions provided the GSA.

Deploy the google-cloud-cli container using the workload-identity-test.yaml manifest.

# workload-identity-test.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: workload-identity-test
  namespace: default
  annotations:
    # disable sidecar
    sidecar.istio.io/inject: "false"
    # disable egress traffic capture, allow access to google metadata server
    traffic.sidecar.istio.io/excludeOutboundIPRanges: "169.254.169.254/32"
spec:
  selector:
    matchLabels:
      app: workload-identity-test
  template:
    metadata:
      labels:
        app: workload-identity-test
    spec:
      serviceAccountName: my-wi-ksa
      containers:
      - name: workload-identity-test
        image: gcr.io/google.com/cloudsdktool/google-cloud-cli:384.0.1
        command: ["sleep","infinity"]

And apply it to the cluster

kubectl apply -f workload-identity-test.yaml

Now use gcloud from inside the container to view the available clusters and VM instances.

# set variables
dep=workload-identity-test
ns=default

# wait for deployment to be ready
kubectl wait deployment -n $ns $dep --for condition=Available=True --timeout=90s

# able to view clusters because GSA has 'roles/container.clusterViewer'
kubectl exec -it deployment/$dep -n $ns -- gcloud container clusters list

# CANNOT view VM instances because GSA does not have 'roles/compute.viewer'
kubectl exec -it deployment/$dep -n $ns -- gcloud compute instances list

As you can see, our container now has access to Google Cloud Services without requiring mounted or hard-coded secrets.  This is why it is the preferred method for secure access to services.

Example Project

If you want to see a full example, use my github project where I go through various method of access including Workload Identity.  Read the README file for full instructions.

# show current default kubeconfig and context
echo $KUBECONFIG
kubectl config current-context

# run menu that will take you through various scenarios
git clone https://github.com/fabianlee/gcloud-kubectl-workload-identity.git
cd gcloud-kubectl-workload-identity.git
./menu.sh

Running the menu will then show you the available deployment scenarios.  Execute each action from top to bottom.

===========================================================================
 MAIN MENU for gke_my-gkeproj1-xxx_us-east1_cluster1
===========================================================================

gke-wi-check     Validate workload identity status of GKE cluster                         

generic          Deploy generic image                                                     
simpleksa        Deploy image running as simple KSA                                       
annotatedksa     Deploy image running as KSA which is annotated with GSA                  
jsonsecret       Deploy image with GCP json secret mounted                                
workloadid       Deploy image running as KSA with mapping to GSA                          

deployments      Validate health of deployments, KSA, GSA role                            

generic-test     Rest generic access to kubectl and gcloud                                
simpleksa-test   Test access to kubectl and gcloud                                        
annotatedksa-test Test access to kubectl and gcloud                                        
jsonsecret-test  Test access to kubectl and gcloud                                        
workloadid-test  Test access to kubectl and gcloud                                        

ksa-can-i        Test KSA permissions using kubectl can-i                                 

teardown         remove deployments, KSA, and KSA/GSA binding                             

Which action (q to quit) ? 

 

REFERENCES

google ref, enable workload identity

gcloud ref, add-iam-policy binding

github fabianlee, example project for testing workload identity