GitLab: Continuous Deployment with Agent for Kubernetes and GitLab pipeline


Categories: DevOps

GitLab pipelines are frequently used for the building of binaries and publishing of images to container registries, but do not always follow through with Continuous Deployment to a live environment.

One reason is that pipelines do not usually have access to the internal systems where these applications are meant to be deployed.

In this article, I will describe how to deploy the GitLab Agent for Kubernetes into a GitLab project and target Kubernetes cluster.  This will grant the pipeline kubectl level access to the Kubernetes cluster, and therefore a way to deploy workloads to a live environment.

GitLab Access Token for registering Agent

We will be making a GitLab API call for registering a new agent.  This could be done from the GitLab web UI, but it provides better automation support to show the API call.

But in order to make the GitLab API call, we need an Access Token that has the correct privilege.  Here are your options, choose one:

We only need ‘read_api’ scope to do group/project name resolution, but registering the agent requires ‘api’.

Personal Access Token

Follow the documentation here and go to Main Page > Click your Avatar > Settings > Edit Profile > Access Tokens and “Add new token”, and use these values:

  • name=create-agent
  • scope=api,k8s_proxy

Project Access Token

Follow the documentation here and from your project repository go to Settings > Access Tokens and “Add new token”, and use these values:

  • name=create-agent
  • scope=api,k8s_proxy
  • role=Maintainer

The ability to create a Project Access Token (when your project is in a group, not in the root namespace) is only available with GitLab Premium, so if you are at the free tier you must fallback to using GitLab Personal Access Token (PAT) to create a Project Agent.

Register Agent with GitLab project

The Agents API provides an automated way for us to register an agent for a GitLab repository.  You should make a personal fork of my Gitlab project “gitlab-agent-for-k8s-manifest” (not use a clone), so you have the privileges to register an Agent.

# required utilities
sudo apt install jq curl git -y

# you should create your own personal fork
git clone https://gitlab.com/gitlab-agent2/gitlab-agent-for-k8s-manifest.git && cd $(basename $_ .git)

# define variables
project_name=$(basename $PWD)
agent_name=agent-${project_name}

# for public gitlab, will differ if you have private instance
GITLAB_URL=https://gitlab.com

# either personal or group Access Token with 'read_api,create_runner' scope
ACCESS_TOKEN=<personal-or-group-AccessToken>

# invoke API to resolve project id
project_id=$(curl --fail -sX GET "$GITLAB_URL/api/v4/projects?search=$project_name" --header "PRIVATE-TOKEN: $ACCESS_TOKEN" | jq ".[] | select (.name==\"$project_name\").id")

# invoke API to register Agent for K8S
echo "about to register Agent '$agent_name' for K8S project '$project_name' with id $project_id"
agent_id=$(curl --fail -sX POST $GITLAB_URL/api/v4/projects/$project_id/cluster_agents --data "{\"name\":\"$agent_name\"}" -H "Content-Type:application/json" --header "Private-Token: $ACCESS_TOKEN" | jq '.id')
echo "agent '$agent_name' created with id $agent_id"

# invoke API to create token for Agent
output=$(curl --fail -sX POST $GITLAB_URL/api/v4/projects/$project_id/cluster_agents/$agent_id/tokens --data "{\"name\":\"mytoken\"}" -H "Content-Type:application/json" --header "Private-Token: $ACCESS_TOKEN")
token_id=$(echo $output | jq '.id')
token_secret=$(echo $output | jq -r '.token')
echo "for agent '$agent_name', new token created with id '$token_id' and secret '$token_secret'"

You must save the value for the token secret because it will only be displayed once and must be used later when registering agents (RUNNER_TOKEN).

Validate registration

If you navigate to your project from the GitLab web UI: Operate > Kubernetes Clusters

The Agent will now be visible, but report “Never connected” because the agent is not yet installed on the Kubernetes cluster side.

Authorize the Agent for your project

You must authorize the Agent to access the project where you keep your Kubernetes manifests. You can authorize the agent to access individual projects or authorize an entire group.

Create the configuration file in the target git project at “.gitlab/agents/<agent-name>/config.yaml”.

echo "going to create authorization file for agent '$agent_name'"

# make authorization config file
mkdir -p .gitlab/agents/$agent_name
cat << EOF > .gitlab/agents/$agent_name/config.yaml
ci_access:
  projects:
    - id: $project_name
EOF

# show authorization file just created
cat .gitlab/agents/$agent_name/config.yaml

# add authorization to git repo
git add .gitlab/agents/$agent_name/config.yaml
git commit -m "added authorization config for $agent_name" .gitlab/agents/$agent_name/config.yaml
git push -o ci.skip

This authorization may take a few minutes to propagate.

Installing Agent on Cluster

Now it is time to deploy the Agent on the Kubernetes cluster using Helm.  The Agent deployed to the cluster initiates the connection back to the GitLab KAS.

# server side URL for public GitLab KAS
KAS_URL=wss://kas.gitlab.com

# bring values over from previous section
project_name=gitlab-agent-for-k8s-manifest
agent_name=agent-${project_name}
token_secret=glagent-xxxxxxxxxxxxxx

# add helm repo
helm repo add gitlab https://charts.gitlab.io && helm repo update gitlab
# deploy release into cluster
helm upgrade --install $agent_name gitlab/gitlab-agent --namespace ${agent_name} --create-namespace --set image.tag=v16.5.0 --set config.token=$token_secret --set config.kasAddress=$KAS_URL

# show release status
helm status $agent_name -n $agent_name
helm history $agent_name -n $agent_name

# show objects just created in cluster (single deployment with 2 pods)
kubectl get all -n $agent_name

Validate agent connection

If you navigate to your project from the GitLab web UI: Operate > Kubernetes Clusters

The Agent will now be visible and report “Connected”.

GitLab pipeline to build/publish/deploy to K8S cluster

From within a GitLab pipeline we are now provided the additional ‘KUBERNETES’ predefined variable that can be used to run kubectl commands against our cluster.

Note there is no requirement for the Kubernetes Cluster to be reachable via a public endpoint, the bi-directional gRPC communication is initiated from the Helm-installed agentk installed on the Cluster.  The pipeline-executed kubectl commands flow through the server-side KAS per the architecture document.

Here is the full .gitlab-ci.yaml, I will highlight a couple of critical sections:

Build and Publish Image

build-and-push-image:
  stage: build
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind
  script: |
    echo "value sourced from build.env is $VERSION"
    # append image version
    export IMAGE_TAG="$CI_REGISTRY_IMAGE:$VERSION"
    # do build and push
    BUILD_TIME=$(date -u '+%Y-%m-%d_%H:%M:%S')
    docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    docker build -f Dockerfile --build-arg MY_VERSION=$VERSION --build-arg MY_BUILDTIME=$BUILD_TIME --build-arg MY_GITREF=$CI_COMMIT_SHORT_SHA -t $IMAGE_TAG .
    docker push $IMAGE_TAG

Deploy workload to Cluster

If the ‘KUBECONFIG’ predefined variable exists, the pipeline goes on to deploy the image to the connected Kubernetes Cluster.

It first sets the kubecontext, then runs the kubectl commands to deploy the latest manifest, then finally a curl against the deployed service.

k8s-agent-test1:
  stage: test
  rules:
    - if: $KUBECONFIG
  image:
    name: bitnami/kubectl:1.27.7-debian-11-r0
    entrypoint: ['']
  variables:
    DEPLOYMENT_NAME: golang-hello-world-web
  before_script: |
    echo "value sourced from build.env is $VERSION"
    export IMAGE_TAG="$CI_REGISTRY_IMAGE:$VERSION"
  script: |
    echo "set kubectl context"
    kubectl config use-context $CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:agent-$CI_PROJECT_NAME

    echo "test namespace fetch"
    kubectl get ns

    echo "create namespace if necessary"
    kubectl get ns $CI_PROJECT_NAME || kubectl create ns $CI_PROJECT_NAME

    echo "deploying manifest to kubernetes cluster with $IMAGE_TAG"
    MY_MESSAGE=$CI_PROJECT_NAME
    cat manifest.yaml | sed "s#MY_IMAGE#$IMAGE_TAG#g; s/MY_MESSAGE/$MY_MESSAGE/g;" | kubectl apply -f - -n $CI_PROJECT_NAME

    echo "waiting for deployment to be ready, then stabilize"
    kubectl rollout status deployment $DEPLOYMENT_NAME -n default --timeout=90s
    sleep 15

    echo ===============
    echo "logs..."
    kubectl logs deployment/$DEPLOYMENT_NAME -n $CI_PROJECT_NAME

    echo ===============
    echo "testing curl against deployment entry point and health endpoint"
    kubectl exec deployment/$DEPLOYMENT_NAME -n $CI_PROJECT_NAME -- wget -q http://localhost:8080/ -O-
    echo ===============
    kubectl exec deployment/$DEPLOYMENT_NAME -n $CI_PROJECT_NAME -- wget -q http://localhost:8080/healthz -O-

Validate from Cluster

You can also validate with direct kubectl calls against your Kubernetes cluster (from a jumpbox where your cluster is reachable).

# setup variables
project_name=gitlab-agent-for-k8s-manifest
deployment_name=golang-hello-world-web

# validate namespace for agent and another for deployment
$ kubectl get ns | grep $project_name
agent-gitlab-agent-for-k8s-manifest Active 3h31m
gitlab-agent-for-k8s-manifest Active 3h20m

# validate deployment of image
$ kubectl get deployment -n $project_name
NAME READY UP-TO-DATE AVAILABLE AGE
golang-hello-world-web 1/1 1 1 3h22m

# validate image used by deployment
$ kubectl get deployment $deployment_name -n $project_name -o=yaml | grep "^\s*image:"

# show environment variables passed to container
$ kubectl get deployment $deployment_name -n $project_name -o=jsonpath="{.spec.template.spec.containers[0].env}" | jq

# show logs for container
kubectl logs deployment/$deployment_name -n $project_name

# curl to endpoint
kubectl exec -it deployment/$deployment_name -n $project_name -- wget -q http://localhost:8080/ -O-
# curl to health endpoint
kubectl exec -it deployment/$deployment_name -n $project_name -- wget -q http://localhost:8080/healthz -O-

Limiting kube-API privileges of Agent

In this article, I went through the most basic Helm installation of the Agent which runs as a Service Account with ‘cluster-admin’ privileges.

To impose limits on the namespaces and objects that the Agent can work on, read my article on applying least privilege to the Agent service account.

 

 

REFERENCES

fabianlee GitLab, source for this article

GitLab, Using GitLab CI/CD with a K8S cluster

GitLab, connecting a Kubernetes cluster with GitLab

GitLab, Agent for Kubernetes architecture

GitLab, installing the Agent for Kubernetes

GitLab, Agent Server for Kubernetes (kas)

GitLab, agent configuration file for GitOps reference

GitLab, upgrade the agent using Helm

GitLab, example source code repo using agent for k8s

GitLab blog, understanding the GitLab Agent for K8S

GitLab Fernando Diaz, deploying the GitLab Agent for K8S with minimal privileges

GitLab, agent for kubernetes source project

mit.edu, cluster agent API reference

GitLab, installing server-side KAS server for Linux

stackoverflow.com, see if group search can be done by exact group name or graphql

harness.io, different deployment strategies (rolling, canary, A/B)

NOTES

GoLang has a concept of build-time variables that can be compiled into the binary, and we use that to set the version, build time, and latest git sha.

You can fetch a single GitLab project with API, but must include the project namespace (root namespace or group name)

project_ns=<owner-or-group-name>
curl --fail -sX GET "$GITLAB_URL/api/v4/projects/${project_ns}%2F${project_name}" --header "PRIVATE-TOKEN: $ACCESS_TOKEN" | jq

Get list of Agents

curl -sX GET $GITLAB_URL/api/v4/projects/$project_id/cluster_agents --header "Private-Token: $ACCESS_TOKEN" | jq

Get detail on Agent

agent_id=xxxxxx # get from listing of all agents
curl -sX GET $GITLAB_URL/api/v4/projects/$project_id/cluster_agents/$agent_id --header "Private-Token: $ACCESS_TOKEN" | jq

List tokens for Agent

curl -sX GET $GITLAB_URL/api/v4/projects/$project_id/cluster_agents/$agent_id/tokens --header "Private-Token: $ACCESS_TOKEN" | jq

Delete token for Agent

token_id=xxxxx # get from listing all tokens
curl -sX DELETE $GITLAB_URL/api/v4/projects/$project_id/cluster_agents/$agent_id/tokens/$token_id --header "Private-Token: $ACCESS_TOKEN"

Delete Agent

curl -sX DELETE $GITLAB_URL/api/v4/projects/$project_id/cluster_agents/$agent_id --header "Private-Token: $ACCESS_TOKEN"

Show Helm values

helm get values $agent_name -n gitlab-agent-$agent_name