With Workload Identity enabled on a GKE cluster, your container can access Google Cloud API services (Compute Engine, Storage, etc.) using a Kubernetes Service Account (KSA).
This is done by having the container run as the KSA, where the KSA has been bound to the Google Service Account (GSA). This is the recommended way of running workloads that require secure access to Google cloud services.
In this article I will show how enable Workload Identity on your GKE cluster and node pool, and then how to run a container image as a KSA that can run gcloud commands because of its binding to a GSA.
Enable Workload Identity
The first step is to enable Workload Identity at the GKE cluster and node pool level. First, check the current settings of the GKE cluster.
# list clusters
export project_id=$(gcloud config get-value project)
gcloud container clusters list --format="csv[no-heading](name,location)"
# set variables
cluster_name=xxxxxx
# use either 'region' or 'zone'
location_flag="--region xxxxx"
# set name of node pool
nodepool_name=$(gcloud container node-pools list --cluster=$cluster_name $location_flag --format="value(name)")
# if empty, then workload identity not set at cluster level
# if workload identity enabled, will return ${project_id}.svc.id.goog
gcloud container clusters describe $cluster_name$location_flag --format="value(workloadIdentityConfig.workloadPool)"
# if empty, then workload identity not set at node pool level
gcloud container node-pools describe $nodepool_name --cluster=$cluster_name $location_flag --format="value(config.workloadMetadataConfig.mode)"
If Workload Identity needs to be enabled, then use the below commands.
# enable at cluster level, can take ~20 minutes
gcloud container clusters update $cluster_name $location_flag --workload-pool=${project_id}.svc.id.goog
# enable at node pool level
# takes time because it does serial rebuild of each node in pool
gcloud container node-pools update $nodepool_name --cluster=$cluster_name $location_flag --workload-metadata=GKE_METADATA
Create Google Service Account (GSA)
Create a Google Service Account “gcloud-user” that has the IAM roles required to access the desired Google services.
GSA=gcloud-user
# create account, allow settling with delay
gcloud iam service-accounts create $GSA --project=$project_id
sleep 15
# get fully qualified email for GSA
# syntax will be ${GSA}@${project_id}.iam.gserviceaccount.com
svcEmail=$(gcloud iam service-accounts list --project=$project_id --filter="name ~ ${GSA}@" --format="value(email)")
# add these roles to GSA
roles="roles/container.clusterViewer roles/container.viewer roles/iam.workloadIdentityUser"
for role in $roles; do
gcloud projects add-iam-policy-binding $project_id --member=serviceAccount:$svcEmail --role=$role > /dev/null
done
echo "show final '$svcEmail' roles"
gcloud projects get-iam-policy $project_id --flatten="bindings[].members" --filter="bindings.members=serviceAccount:$svcEmail" --format="value(bindings.role)"
Create Kubernetes Service Account (KSA)
Now create the KSA with an annotation that references the GSA. This can be done by applying the my-wi-ksa.yaml manifest with kubectl.
# my-wi-ksa.yaml
---
# binds k8s service account to role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ksa-wi-reader-binding
subjects:
- kind: ServiceAccount
name: ${KSA}
namespace: default
roleRef:
kind: ClusterRole
name: ksa-wi-reader
apiGroup: rbac.authorization.k8s.io
---
# role that defines permissions
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ksa-wi-reader
rules:
- apiGroups: [""]
resources: ["namespaces","nodes", "pods", "secrets", "services", "configmaps","crontabs","persistentvolumes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["daemonsets", "replicasets", "statefulsets"] # intentionally left out 'deployments'
verbs: ["get", "list", "watch"]
---
# k8s service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: ${KSA}
namespace: default
annotations:
iam.gke.io/gcp-service-account: gcloud-user@${project_id}.iam.gserviceaccount.com
Then apply to the cluster.
# envsubst used to replace with env values export KSA=my-wi-ksa envsubst < my-wi-ksa.yaml | kubectl apply -f -
Bind KSA and GSA
With the GSA and KSA now created, we can bind them together so that the KSA can impersonate the GSA roles.
# bind GSA and KSA
gcloud iam service-accounts add-iam-policy-binding $svcEmail --role roles/iam.workloadIdentityUser --member serviceAccount:${project_id}.svc.id.goog[default/$KSA]
Create Container running as KSA
The final step is to deploy a container running as the KSA. From inside this container, the Google Cloud API will be available with the level of permissions provided the GSA.
Deploy the google-cloud-cli container using the workload-identity-test.yaml manifest.
# workload-identity-test.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: workload-identity-test
namespace: default
annotations:
# disable sidecar
sidecar.istio.io/inject: "false"
# disable egress traffic capture, allow access to google metadata server
traffic.sidecar.istio.io/excludeOutboundIPRanges: "169.254.169.254/32"
spec:
selector:
matchLabels:
app: workload-identity-test
template:
metadata:
labels:
app: workload-identity-test
spec:
serviceAccountName: my-wi-ksa
containers:
- name: workload-identity-test
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:384.0.1
command: ["sleep","infinity"]
And apply it to the cluster
kubectl apply -f workload-identity-test.yaml
Now use gcloud from inside the container to view the available clusters and VM instances.
# set variables dep=workload-identity-test ns=default # wait for deployment to be ready kubectl wait deployment -n $ns $dep --for condition=Available=True --timeout=90s # able to view clusters because GSA has 'roles/container.clusterViewer' kubectl exec -it deployment/$dep -n $ns -- gcloud container clusters list # CANNOT view VM instances because GSA does not have 'roles/compute.viewer' kubectl exec -it deployment/$dep -n $ns -- gcloud compute instances list
As you can see, our container now has access to Google Cloud Services without requiring mounted or hard-coded secrets. This is why it is the preferred method for secure access to services.
Example Project
If you want to see a full example, use my github project where I go through various method of access including Workload Identity. Read the README file for full instructions.
# show current default kubeconfig and context echo $KUBECONFIG kubectl config current-context # run menu that will take you through various scenarios git clone https://github.com/fabianlee/gcloud-kubectl-workload-identity.git cd gcloud-kubectl-workload-identity.git ./menu.sh
Running the menu will then show you the available deployment scenarios. Execute each action from top to bottom.
=========================================================================== MAIN MENU for gke_my-gkeproj1-xxx_us-east1_cluster1 =========================================================================== gke-wi-check Validate workload identity status of GKE cluster generic Deploy generic image simpleksa Deploy image running as simple KSA annotatedksa Deploy image running as KSA which is annotated with GSA jsonsecret Deploy image with GCP json secret mounted workloadid Deploy image running as KSA with mapping to GSA deployments Validate health of deployments, KSA, GSA role generic-test Rest generic access to kubectl and gcloud simpleksa-test Test access to kubectl and gcloud annotatedksa-test Test access to kubectl and gcloud jsonsecret-test Test access to kubectl and gcloud workloadid-test Test access to kubectl and gcloud ksa-can-i Test KSA permissions using kubectl can-i teardown remove deployments, KSA, and KSA/GSA binding Which action (q to quit) ?
REFERENCES
google ref, enable workload identity
gcloud ref, add-iam-policy binding
github fabianlee, example project for testing workload identity