GCP: LDAP authentication for Anthos VMware clusters using Anthos Identity Service


Categories: Hyperscaler, Kubernetes

Anthos Identity Service allows an organization to tie into their existing Identity Provider to authenticate and authorize users into their Anthos clusters.

In this article, I will show how the authentication for an Anthos on VMware cluster can be integrated into an existing Active Directory deployment, and further how a user’s AD group membership can determine RBAC roles within the cluster.

Prerequisites

  • Deployment of Anthos on VMware on-premise cluster
  • Deployment of Active Directory with exposed secure LDAP (port 636).  CA cert if not public
  • LDAP service account and credentials for binding

I have written previous articles on installing a Windows 2019 Domain Controller, and installing Anthos on VMware 1.13.

LDAP validation

It can save a lot of time if you validate your LDAP connectivity and hierarchy, instead of waiting to test in-cluster.  We can use the ldapsearch utility on Ubuntu to prove out our connectivity and filters.

sudo apt install ldap-utils -y

If the CA certificate of the Windows Domain Controller is self-signed (not from a public authority), then place the CA file (in PEM format) locally and set this environment variable so it can be used to validate the chain of trust.

export LDAPTLS_CACERT=adfsCA.pem

Then setup the basic variables for connecting to AD.  Tailor to your environment, these are based off my previous article on installing a Windows 2019 Domain Controller and setting up a suite of test users using Powershell.

ldap_server=win2k19-dc1.fabian.lee
base_DC="DC=FABIAN,DC=LEE"

# Anthos Identity Service expects full DN (not just id@domain)
binding_user="CN=ldap1,CN=Users,${base_DC}"
binding_pass="ThisIsMyP4ss!"

Test connectivity using service account

Now test a basic search that returns the top level containers in AD using the service account binding credentials.

ldapsearch -LLL -H ldaps://${ldap_server}:636 -D $binding_user -w $binding_pass -b $base_DC -s one dn

If this does not work, do not move forward until you figure out if this is a credentials issue, network connectivity, or CA cert trust validation.

Test the base user filter

This will list all ‘user’ objects in CN=Users

ldapsearch -LLL -H ldaps://${ldap_server}:636 -D $binding_user -w $binding_pass -b CN=Users,$base_DC -s one "(objectClass=user)" sAMAccountName

We will use the short ‘sAMAccountName’ attribute of the user object for login (e.g. engineer1), and ‘userPrincipalName’ (e.g. engineer1@fabian.lee) for any user specific subjects.

# short login id
ldapsearch -LLL -H ldaps://${ldap_server}:636 -D $binding_user -w $binding_pass -b CN=Users,$base_DC -s one "(objectClass=user)" sAMAccountName

# RBAC role represented with unique, fully qualified domain
ldapsearch -LLL -H ldaps://${ldap_server}:636 -D $binding_user -w $binding_pass -b CN=Users,$base_DC -s one "(objectClass=user)" userPrincipalName

Test the base group filter

This will list all ‘group’ objects in CN=Users

ldapsearch -LLL -H ldaps://${ldap_server}:636 -D $binding_user -w $binding_pass -b CN=Users,$base_DC -s one "(objectClass=group)" cn

Armed with these values and filters we can now continue, confident that our cluster config settings are valid.

Configure Kubernetes cluster

Verify Anthos Identity Service deployment

Verify that Anthos Identity Service components are available on this cluster.  They come standard for Anthos on-prem installations.

# namespace should exist
kubectl get ns anthos-identity-service

# deployment of AIS should exist
kubectl get deployment -n anthos-identity-service ais

# view logs for deployment (ok to have 401 and UNAVAILABLE errors right now)
kubectl logs -n anthos-identity-service deployment/ais -c ais-container

Create Cluster RBAC roles

In anticipation of Cluster authorization based on a user’s membership in an Active Directory group, we are going to setup RBAC (Role-based Access Control) for two Active Directory groups:

  • engineers
  • managers

These are groups that we created in a previous article using a Powershell script, we will put in place the following rules:

  • Users in the ‘engineering’ group can get/watch/list pods in any namespace (ClusterRole/ClusterRoleBinding), but cannot interact with any other objects (e.g. deployments, daemonsets, secrets, etc)
  • Users in the ‘managers’ group can get/watch/list pods and secrets but only in the default namespace (Role/RoleBinding), and not in any other namespace
# download roles and bindings from my github project
project_url=https://raw.githubusercontent.com/fabianlee/anthos-nested-esx-manual/main

# ClusterRole for 'engineers' group
wget $project_url/anthos-identity/pod-reader-clusterrole.yaml
wget $project_url/anthos-identity/pod-reader-clusterrolebinding.yaml

# Role for 'managers' group
wget $project_url/anthos-identity/defaultns-reader-role.yaml
wget $project_url/anthos-identity/defaultns-rolebinding.yaml

# apply engineers RBAC at cluster level
kubectl apply -f pod-reader-clusterrole.yaml
kubectl apply -f pod-reader-clusterrolebinding.yaml

# apply managers RBAC at namespace level
kubectl apply -f defaultns-reader-role.yaml
kubectl apply -f defaultns-rolebinding.yaml

This has no effect when logging in use the default kubeconfig with client-key-data for authentication, but in later sections when users are forced to go through LDAP for authentication (they will never be given private certs), their AD group membership will determine what operations are possible.

Create LDAP service account secret

Create a secret in the ‘anthos-identity-service’ namespace that represents the LDAP service account.  Per my comment in the previous section, the binding user must be the fully qualified DN of the LDAP service account (not just id@domain) because that is what Anthos Identity Server demands.

kubectl create -f - << EOF
apiVersion: v1
kind: Secret
metadata:
  name: ldap-secret
  namespace: anthos-identity-service
type: kubernetes.io/basic-auth
stringData:
  username: $binding_user # e.g. CN=xx,CN=Users,DC=x,DC=y
  password: $binding_pass
EOF

Patch ClientConfig object

Load the custom CA pem into a single-lined base64 variable (if Domain Controller CA cert is not public).

ca_base64=$(cat adfsCA.pem | base64 | tr -d '/\n//')

Then create the ClientConfig manifest patch that we can apply to the cluster.

cat <<EOF > clientconfig-patch.yaml
kind: ClientConfig
metadata:
  name: default
  namespace: kube-public
spec:
  authentication:
  - name: ldap
    ldap:
      host: ${ldap_server}:636
      certificateAuthorityData: $ca_base64
      connectionType: ldaps
      serviceAccountSecret:
        name: ldap-secret
        namespace: anthos-identity-service
        type: basic
      user:
        baseDN: CN=Users,$base_DC
        filter: (objectClass=user)
        identifierAttribute: userPrincipalName
        loginAttribute: sAMAccountName
      group:
        baseDN: CN=Users,$base_DC
        filter: (objectClass=group)
        identifierAttribute: cn
EOF

Note that the ldap attribute names are case-sensitive,  e.g. if you see ‘userPrincipalName’ in ADSI Edit of the Domain Controller, then you must use this same case in the yaml above.

Apply this patch to the ClientConfig object of the cluster.

kubectl -n kube-public patch clientconfig/default --type=merge --patch-file clientconfig-patch.yaml

And now the logs should show messages indicating “LDAP[0] started”.

kubectl logs -n anthos-identity-service deployment/ais -c ais-container -f

Validate User access

Now let’s see what this looks like from an end user perspective, i.e. a cluster developer/deployer.

Install the Google Cloud CLI with Anthos components

Follow the instructions on the official docs for installing gcloud and its Anthos components.  On Ubuntu this can be done by adding the google apt repository.

sudo apt install -y apt-transport-https ca-certificates gnupg curl

# add key and repo
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo tee /usr/share/keyrings/cloud.google.asc
chmod 666 /usr/share/keyrings/cloud.google.asc
echo "deb [signed-by=/usr/share/keyrings/cloud.google.asc] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

# install components
sudo apt update
sudo apt install google-cloud-cli google-cloud-sdk google-cloud-sdk-anthos-auth kubectl -y

# initialize settings
gcloud init

Generate Anthos login config (one time)

Generate the Anthos login config file with gcloud.

# create ldap config file 'kubectl-anthos-config.yaml'
gcloud anthos create-login-config --kubeconfig=$KUBECONFIG

The cluster name used at ‘.spec.name’ will be used later for ‘gcloud anthos auth login’

sudo apt install yq -y
# set variable 'cluster_name' for use later
cluster_name=$(yq eval '.spec.name' kubectl-anthos-config.yaml)
echo "cluster name is $cluster_name"

Move the login config to the expected default location.

mkdir -p  $HOME/.config/google/anthos/
mv kubectl-anthos-config.yaml $HOME/.config/google/anthos/kubectl-anthos-config.yaml

Note: ‘kubectl-anthos-config.yaml’ does not remove the need for a KUBECONFIG by end-users, this config is used to augment their KUBECONFIG.

You can safely remove the ‘group’, ‘serviceAccountSecret’, and ‘user’ sections from this file, these are values populated from the ClientConfig that are not necessary on the client side.

KUBECONFIG before modification

The KUBECONFIG used to generate the ‘kubectl-anthos-config.yaml’ login config above is the full-permission version containing the ‘client-key-data’ that provides immediate privileged access to the cluster.

Note the contexts and users currently contained in the KUBECONFIG.

kubectl config get-contexts
kubectl config get-users
kubectl config get-clusters

These contain the escalated privileges and private client-key-data that we would NOT want to provide end users.  We will remove these in a later section to prepare for end-user distribution.

LDAP authentication using login config

Now we will use the KUBECONFIG coupled with ‘kubectl-anthos-config.yaml’ to initiate the LDAP login flow.  Per our ClientConfig, we are using ‘sAMAccountName’ as the attribute for our user login, so we use the simple name ‘engineer1’ below without a domain suffix.

# no need to specify 'login-config' flag since file is in default location
# cluster name is from anthos login config
$ gcloud anthos auth login --cluster=$cluster_name

Please enter the ldap user for [ldap] on cluster [user1]: engineer1
Please enter the ldap password for [ldap] on cluster [user1]: 
Configuring Anthos authentication 
2022/10/18 10:58:31 LDAP login successful.
Re-login after the token in the kubeconfig expires in 1 hr(s).
Configuring Anthos authentication success.

The KUBECONFIG has now been modified with an additional cluster, context, and user with ‘token’ value.

The last line in the deployment log should indicate “Kubernetes webhook adapter successfully authenticated request”.

$ kubectl logs -n anthos-identity-service deployment/ais -c ais-container -f

KUBECONFIG prepared for end-user distribution

As mentioned above, the ‘gcloud anthos auth login’ from the previous section modified our KUBECONFIG.  There is now an additional cluster, context and user entry for our LDAP access.

echo "cluster name is $cluster_name"

# should now have extra cluster entry '$cluster_name'
kubectl config get-clusters

# should now have extra context '$cluster_name-$cluster_name-anthos-default-user'
kubectl config get-contexts

# should now have extra user with '$cluster_name-anthos-default-user'
kubectl config get-users

But since we do not want the original escalated permissions and client-key-data to be provided to end-users, we can remove these to create a minimal KUBECONFIG that can be safely distributed.

kubectl config delete-context <originalContext>
kubectl config delete-user <originalUser>
kubectl config delete-cluster <originalCluster>

The KUBECONFIG should now only contain the essential information required to access the cluster with LDAP authentication.  Clear your current token value before distribution, it will be regenerated when the end-users do ‘gcloud anthos auth login’.

# minimal, without any escalated privileges or private keys
cat $KUBECONFIG

# clear your token value
sed -i 's/token: .*/token: /' $KUBECONFIG

It is this minimal KUBECONFIG paired with the ‘kubectl-anthos-config.yaml’ that will be distributed to end-user operators/developers/deployers.

Validate ‘engineers’ group access

Remember from the RBAC configuration section above, that the ‘engineers’ group has a ClusterRole to read pods from any namespace.  But it does not have access to any other objects.

# success
$ kubectl get pods -n default

# success
$ kubectl get pods -n kube-system

# informs you that trying to pull deployments from different namespace will fail
$ kubectl auth can-i get deployments -n default
no

# FAILS! because ClusterRole only grants permissions on 'pods'
$ kubectl get deployments -n default
Error from server (Forbidden): deployments.apps is forbidden: User "engineer1@test.local" cannot list resource "deployments" in API group "apps" in the namespace "default"

Validate ‘managers’ group access

Now let’s switch over to user ‘manager1’ which is a member of the ‘managers’ group.  This group has a namespace bound Role that allows it to see pods and secrets, but only in the default namespace.

$ gcloud anthos auth login --cluster=user1
Please enter the ldap user for [ldap] on cluster [user1]: manager1
Please enter the ldap password for [ldap] on cluster [user1]: 
Configuring Anthos authentication 
2022/10/18 12:34:07 LDAP login successful.
Re-login after the token in the kubeconfig expires in 1 hr(s).
Configuring Anthos authentication success.

Let’s test the ‘managers’ RBAC permissions which limit it to the default namespace.

# success
$ kubectl get pods -n default

# success
$ kubectl get secrets -n default

# informs you that trying to pull pods from different namespace will fail
$ kubectl auth can-i get pods -n kube-system
no

# FAILS! because Role only grants permission in default namespace
$ kubectl get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "manager1@test.local" cannot list resource "pods" in API group "" in the namespace "kube-system"

 

 

REFERENCES

google ref, Anthos Identity Service with LDAP

google ref, setup an LDAP provider for Anthos Identity Service

google ref, setup user access for Anthos Identity Service

google ref, install the Google Cloud CLI with Anthos components

fabianlee.org, using ldapsearch

man page, ldapsearch

google ref, gcloud anthos create-login-config

theitbros.com, using ldapsearch on active directory

NON-GKE REFERENCES

kubernetes.io, points out that CA cert check is strict and CA:true must be set

Pratishtha Agarwal, securely authenticate to gke anthos cluster with okta

docs.bitnami, applying RBAC to K8S cluster to limit namespace access

Mercy Kemei, Active Directory auth for Kubernetes

Mercy Kemei, authenticate K8S dashboard users with Active Directory

NOTES

ldapsearch needs the CA cert in pem format (not binary DER), if conversion is necessary use openssl.

openssl x509 -in myCA.der -inform DER -out myCA.pem