GCP: Cloud Run with build trigger coming from remote GitHub repository


Categories: Containers, Development, Hyperscaler

GCP build triggers can easily handle Continuous Deployment (CD) when the source code is homed in a Google Cloud Source repository.  But even if the system of record for your source is a remote GitHub repository, these same type of push and tag events can be consumed if you configure a connection and repository link.

Once your GitHub repository is linked, you can then create a build trigger based on an git event such as a push or tag to implement a Continuous Deployment pipeline.

GCP Cloud Run service manually deployed

Let’s first manually deploy the GCP Cloud Run service as a baseline. We will not yet connect it to any Continuous Deployment triggers until later in the article.

I went into detail in a previous article, but below are the summarized steps.

# make sure essential OS packages are installed
sudo apt-get install software-properties-common python3 python3-dev python3-pip python3-venv make curl git -y

# download project
git clone https://github.com/fabianlee/gcp-cloudrun-python3-gunicorn.git
cd gcp-cloudrun-python3-gunicorn

# enable GCP project level services
export PYTHONWARNINGS="ignore:Unverified HTTPS request"
gcloud services enable cloudbuild.googleapis.com artifactregistry.googleapis.com run.googleapis.com

# setup variables
app_name="gcp-cloudrun-python3-gunicorn"
region=$(gcloud config get compute/region)
build_region=us-central1
project_id=$(gcloud config get project)
project_number=$(gcloud projects list --filter="id=$project_id" --format="value(projectNumber)")

# deploy to GCP Cloud Run
gcloud run deploy $app_name --source=. --region=$region --ingress=all --allow-unauthenticated --execution-environment=gen2 --no-use-http2 --quiet

# show details of deployment
gcloud run services describe $app_name --region=$region

# test pull of content
run_url=$(gcloud run services describe $app_name --region=$region --format='value(status.url)')
echo "CloudRun app at: $run_url"
curl $run_url

From the GCP web Console > Cloud Run, you should see a service deployed similar to below.

With this manual Cloud Run service deployed and tested, we will now move on to connecting this remote GitHub repository to a GCP Cloud Build trigger.  The trigger can rebuild and redeploy the Cloud Run service when there are git events.

GCP Cloud Build region restriction

It is important to note that Cloud Build is restricted to certain regions as documented here.  For example, in North America you are limited to placing Cloud Build components in us-central1 or us-west2 only.

It is perfectly acceptable for your Cloud Run instance or Google Artifact Registry to be located in any region, but the Cloud Build specific components need to live in one of the listed regions.  For example, Cloud Build related objects located in us-central1 are fine with pushing the build image to us-east1 and redeploying the new Cloud Run instance to us-east1.

For this article, our ‘build_region’ variable is set to us-central1, while our ‘region’ variable is set to ‘us-east1’ which is where our Cloud Run instance and Google Artifact registry will be homed.

Create GitHub Host connection programmatically

Establishing a connection between GCP and the remote GitHub repository requires creating a GitHub host connection, follow the subsections below.

(GCP) Enable additional API services and permissions

We need to enable additional Google API services and add IAM roles to the Cloud Build related service accounts as a prerequisite.

# enable additional services
gcloud services enable cloudbuild.googleapis.com secretmanager.googleapis.com sourcerepo.googleapis.com

# additional IAM role for Cloud Build Service Agent svc acct
service_account="service-$project_number@gcp-sa-cloudbuild.iam.gserviceaccount.com"
gcloud projects add-iam-policy-binding $project_id --member=serviceAccount:$service_account --role=roles/secretmanager.secretAccessor > /dev/null

# additional IAM roles for Cloud Build svc acct
service_account="$project_number@cloudbuild.gserviceaccount.com"
for role in roles/run.admin roles/iam.serviceAccountUser roles/secretmanager.secretAccessor; do
  gcloud projects add-iam-policy-binding $project_id --member=serviceAccount:$service_account --role=$role > /dev/null
done

(GitHub) Create GitHub PAT

Although you can use manual OAuth2 to establish the connection as your personal/robot user,  per the documentation you can also use a Github Personal Access Token (PAT) which we will prefer for security.

  • Login to github.com, and login
  • Go to Settings > Developer Settings > Personal Access Tokens > Tokens (classic)
  • Click on “Generate New Token (classic)“
  • Fill in values
    • Note=”mygithub”
    • Expiration=30 days
    • Click on “repo” checkbox (which checks all its subitems)
    • Click on single checkbox “user > read:user”
    • Press “Generate token”
  • Copy the PAT value displayed (it cannot be shown again)

This value is the <GITHUB-PAT> value used in a later section when creating a GCP secret.

(GitHub) Install Cloud Build app into GitHub

From the same browser logged in to github.com

  • Go to the Cloud Build installation page in your browser, https://github.com/apps/google-cloud-build
  • Press “Install” button
  • Select scope of application
    • Check “Only select repositories”
    • Select your single repository from the pulldown
    • Press “Install” button
  • You will redirected to a Google login page, login with the Google GCP user that will own the Cloud Run app

Before you move on, make sure you get the application ID of this GitHub application.

  • From github, go to settings > Integrations > Applications
  • One of the applications listed should be “Google Cloud Build”
  • Right-click on the “Configure” button and copy the URL

The copied URL will have a syntax like: https://github.com/settings/installations/xxxxxxx.   The number at the end is the “app-installation-id” used in a later section when creating the GitHub connection.

(GCP) Create GCP secret from PAT

Create a GCP secret using the GitHub PAT generated earlier.

# GITHUB-PAT value from earlier section
echo -n <GITHUB-PAT> | gcloud secrets create my-github-pat --locations=$build_region --replication-policy=user-managed --data-file=-

# show secret full path
gcloud secrets describe my-github-pat

(GCP) Create GCP Host Connection to GitHub

Create the GCP GitHub host connection using the GCP secret.

# app-installation-id from earlier section
github_app_id=xxxxxx
github_conn=mygithub

# create host connection
gcloud alpha builds connections create github $github_conn --authorizer-token-secret-version=projects/$project_number/secrets/my-github-pat/versions/1 --app-installation-id=$github_app_id --region=$build_region

# show connection
gcloud alpha builds connections describe $github_conn --region=$build_region

Link Repository

Use the newly created GitHub Host Connection to link in our single remote GitHub repository.

gcloud alpha builds repositories create $app_name --remote-uri=https://github.com/fabianlee/gcp-cloudrun-python3-gunicorn.git --connection=$github_conn --region=$build_region

# show link
gcloud alpha builds repositories describe $app_name --connection=$github_conn --region=$build_region

If you go to GCP Cloud Build > Repositories (2nd Gen), you should be able to see a Host Connection and link to the remote GitHub repository like below.

Create Artifact Registry repo for Docker images

To provide a Docker repository for the builds, we will create a Google Artifact Registry docker repo.

gar_name=my-gar-repo
gcloud artifacts repositories create $gar_name --repository-format=docker --location=$region

# show repo details
gcloud artifacts repositories describe my-gar-repo --location=$region

Create Build triggers for Continuous Deployment

With the GitHub host connection and link now established, we can create the GCP Build triggers that consume events coming from our remote GitHub repository.

Create a trigger event for ‘git push’ on main branch of git repo.  This uses the gcp-cloudbuild-push.yaml file which defines the continuous deployment steps for rebuilding and redeploying the Cloud Run service.

# create trigger for push using build file
gcloud beta builds triggers create github --name=mytrigger-push --region=$build_region --branch-pattern=^main$ --build-config=gcp-cloudbuild-push.yaml --repository=projects/$project_id/locations/$build_region/connections/$github_conn/repositories/$app_name

# show trigger
gcloud beta builds triggers describe mytrigger-push --region=$build_region

Create a trigger event for ‘git tag’ if tag looks like semantic version (x.y.z).  This uses the gcp-cloudbuild-tag.yaml file which defines the continuous deployment steps for rebuilding and redeploying the Cloud Run service.

# create trigger for tag using build file
gcloud beta builds triggers create github --name=mytrigger-tag --region=$build_region --tag-pattern='\d*\.\d*\.\d*' --build-config=gcp-cloudbuild-tag.yaml  --repository=projects/$project_id/locations/$build_region/connections/$github_conn/repositories/$app_name

# show trigger
gcloud beta builds triggers describe mytrigger-tag --region=$build_region

From the GCP web UI console > Cloud Build > Triggers you can see the triggers just created.

Trigger from a git push

# create event by doing push
echo "" >> README.md; git commit -m "push event" README.md; git push

# will see build status WORKING when queued and executing
gcloud beta builds list --region=$build_region --ongoing

# when done, will be listed as status SUCCESS
gcloud beta builds list --region=$build_region

# show latest hash
$ git log --oneline | head -n1
c5e49dc push event

# curl to the service will show git hash in output
$ curl $run_url
git-c5e49dc Hello, World! called as: /

Trigger from tagging

# create event by doing tag that looks like semantic version
newtag=2.4.1
git tag $newtag && git push origin $newtag

# will see build status WORKING when queued and executing
gcloud beta builds list --region=$build_region --ongoing

# when done, will be listed as status SUCCESS
gcloud beta builds list --region=$build_region

# show latest tag
$ git log --pretty="%h %d %s" | head -n1
c5e49dc (HEAD -> main, tag: 2.4.1, origin/main) push event

# curl to the service will show git tag in output
$ curl $run_url
TAG2.4.1 Hello, World! called as: /

Artifact Registry, docker image repository

The Docker images generated by Cloud Build and used by Cloud Run can be found at the GCP web UI console > Artifact Registry under the ‘my-gar-repo’ repository similar to below.

 

REFERENCES

fabianlee github project

google web UI, Cloud Source

google console UI, Cloud Build

google ref, Cloud Build cloudconfig schema

gcloud ref, gcloud logging read

google ref, Deploying Cloud Run continuous deployment

google issuetracker, why build fails in some gcp regions, only a select few have cloud build enabled

stackoverflow, list of gcp region where cloudrun triggers are valid

google ref, Creating and manging triggers

google ref, Cloud Build service account, <projectId>@cloudbuild.gserviceaccount.com

google ref, IAM roles and permissions

google ref, Cloud Build related service accounts

google ref, Cloud Build configure access to resources

stackoverflow, cloudbuild.yaml submitted with ‘gcloud builds submit’

Vikram Shinde, full example of GCP Cloud Run with CI//CD

Google ref, mirroring a github repository

Google ref, Automating builds connecting to a github repo

Timothy, continuous deployment to Cloud Run on GKE

github GoogleCloudPlatform, Cloud Build triggers for github

google ref, create Google Artifact Repository GAR

stackoverflow, generating multiple tags with GCP trigger cloudbuild yaml

Deepak Kumar Gunjetti, CD with Cloud Run

stackoverflow, github apps installation ID

 

NOTES

Dockerfile image cannot be used alone because ‘build.images[]’ will not be populated

# create trigger using Dockerfile
gcloud beta builds triggers create github --name=mytrigger --region=$build_region --branch-pattern=^main$ --dockerfile-dir="" --dockerfile=Dockerfile --dockerfile-image='us-east1-docker.pkg.dev/$PROJECT_ID/my-gar-repo/$REPO_NAME:$COMMIT_SHA' --repository=projects/$project_id/locations/$build_region/connections/$github_conn/repositories/$app_name

override for values in cloudbuild.yaml using ‘substitutions’ flag

gcloud beta builds triggers create github --name=mytrigger-push --region=$build_region --branch-pattern=^main$ --build-config=gcp-cloudbuild-push.yaml --substitutions "_GAR_REGION=$artifact_region,_GAR_REPO=$gar_name" --repository=projects/$project_id/locations/$build_region/connections/$github_conn/repositories/$app_name

 

Create GitHub Host Connection with manual OAuth2

You can create a GitHub Host Connection using OAuth2 with a user (or robot user) per the official instructions

Initial Browser steps

visit https://console.cloud.google.com, Sign Out of any additional profiles

visit github.com, login as owner of repository

CLI step

# must be GCP Build enabled region (us-east1 is not)
build_region=us-central1

github_conn=mygithub
gcloud alpha builds connections create github $github_conn --region=$build_region

Final Browser step

paste the URL provided in output above into browser.

Browser will report “Connection configured successfully”. You may close this page now.