Terraform: migrate state from local to remote Google Cloud Storage bucket and back


Categories: DevOps, Hyperscaler

In this article I will demonstrate how to take a Terraform configuration that is using a local state file and migrate its persistent state to a remote Google Cloud Storage bucket (GCS).  We will then perform the migration again, but this time to bring the remote state back to a local file.

We will illustrate this concept with a Terraform configuration that creates a single Google GCP VM instance.

Overview

  • Create GCP service account that will perform Terraform actions
  • Create Google Cloud Storage bucket where remote state can be persisted
  • Create GCP VM instance with state saved locally
  • Migrate Terraform state from local to remote Google Cloud Storage bucket
  • Migrate Terraform state from remote Google Cloud Storage bucket to local

Prerequisites

You will need a GCP account, and gcloud and Terraform installed in order to run the steps in this article.

Create GCP service account

Our first step will be to login to GCP with our personal privileges (typically ‘Editor‘ role), that has the ability to create a GCP service account.  This new GCP service account “tf-creator1” will be the one that performs all Terraform actions.

# login with personal user privileges
gcloud config set pass_credentials_to_gsutil true
gcloud auth login

# explicitly set project id based on available list
gcloud config projects list
project_id=<fromList>
gcloud config set project $project_id

newServiceAccount="tf-creator1"

# create service account
gcloud iam service-accounts create $newServiceAccount --display-name "terraform" --project=$project_id

# get email identifier for service account
accountEmail=$(gcloud iam service-accounts list --project=$project_id --filter=$newServiceAccount --format="value(email)")

# download json private key
gcloud iam service-accounts keys create serviceaccount.json --iam-account $accountEmail

# assign IAM roles
for role in roles/compute.admin; do 
  gcloud projects add-iam-policy-binding $project_id --member=serviceAccount:$accountEmail --role=$role > /dev/null
done

# show all IAM roles for service account
gcloud projects get-iam-policy $project_id --flatten='bindings[].members' --filter="bindings.members:serviceaccount:${accountEmail}" --format='value(bindings.role)'

Create GCS bucket

Then we create a remote GCS bucket where terraform state can be persisted.  We must add both the ObjectCreator and ObjectAdmin privileges for the service account.

# create GCS bucket
random_string=$(head /dev/urandom | tr -dc a-z0-9 | head -c 20)
bucket_name="$project_id-$random_string"
gsutil mb -p $project_id gs://$bucket_name

# add self as admin
my_user=$(gcloud config get account)
gsutil iam ch user:${my_user}:admin gs://$bucket_name

# add service account in creator,admin role for creation and deletion
sa_name="$newServiceAccount@${project_id}.iam.gserviceaccount.com"
gsutil iam ch serviceAccount:${sa_name}:objectCreator,objectAdmin gs://$bucket_name

Use Terraform to create GCP VM instance with local state

Clone my github project that has Terraform code to create a GCP VM instance using the earlier downloaded “serviceaccount.json” as credentials, and Terraform state saved locally.

# check git project repo
sudo apt update && sudo apt install -y git
git clone https://github.com/fabianlee/tf-gcp-migrate-state.git

# copy GCP service account key into project directory
cp serviceaccount.json tf-gcp-migrate-state/.
cd tf-gcp-migrate-state 
# init Terraform for local state
tf init
tf plan

# export projectId with special 'TF_VAR_' prefix, so variable is always passed to 'terraform apply'
export TF_VAR_project_id=$(gcloud config get project)

# show backend in use is local
cat backend.tf
# create GCP VM instance and save state locally
tf apply -auto-approve

# confirm local state file is populated
ls -l terraform.tfstate

# show initial tags saved in local state: foo,bar
cat terraform.tfstate | jq ".resources[].instances[].attributes.tags"

# show initial tags on VM instance
gcloud compute instances describe test123 --zone=us-central1-a --format='value(tags)'

Local state is the default if left empty, but note that the state is explicitly local because of our backend definition.

$ cat backend.tf
terraform {
  backend "local" {}
}

Migrate local state to remote GCS bucket

First, modify the backend file to use GCS instead of local storage.

# change backend to use gcs bucket
cp backend.gcs.hcl backend.tf

$ cat backend.tf
terraform {
  backend "gcs" {
    prefix = "tfstate"
    credentials = "serviceaccount.json"
  }
}

Then use terraform init to migrate the state to a remote GCS bucket.

# migrate state to gcp bucket
terraform init -backend-config="bucket=$bucket_name" -migrate-state

# validate tags in gcs remote state are still: foo,bar
terraform state pull | jq ".resources[].instances[].attributes.tags"

# changes tags
TF_VAR_additional_tags='["state-remotegcs"]' terraform apply -auto-approve

# verify remote state now has 'state-remotegcs'
terraform state pull | jq ".resources[].instances[].attributes.tags"

Migrate remote GCS bucket to local state

Modify the backend file to use local state then use the “terraform init” command to migrate the state back to local.

# change backend to use local provider
cp backend.local.hcl backend.tf
cat backend.tf

# migrate state locally
terraform init -migrate-state

# validate local state still shows tag 'state-remotegcs'
cat terraform.tfstate | jq ".resources[].instances[].attributes.tags"

# changes tags
TF_VAR_additional_tags='["state-local"]' terraform apply -auto-approve

# verify local state now has 'state-local' (and not 'state-remotegcs')
cat terraform.tfstate | jq ".resources[].instances[].attributes.tags"

Teardown GCP infrastructure

# destroy GCP VM
terraform destroy -auto-approve

# remove GCS bucket
gsutil rm -fr gs://$bucket_name

 

REFERENCES

fabianlee github, project code

devcoops.com, migrate tf state from remote to local using ‘tf state pull’

medium.com martin.hrasek, migrating tf state between various backends

hashicorp, backend configurations

Brendan Thompson, tf backend dynamically

google, impersonate service account in terraform code

terraform, special ‘TF_VAR_’ variables that get passed to terraform as variables

registry.terraform.io, google provider reference

terraform, resource google compute_instance

google, store terraform state in google cloud bucket

google, IAM roles for GCS bucket storage

gsutil mb – create bucket

gsutil rm – delete bucket

terraform google_compute_instance

NOTES

If gcloud throws ‘InsecureRequestWarning: Unverified HTTPS request is being made to host’

# try to mute like this
export PYTHONWARNINGS="ignore:Unverfied"
# but may need to use broad scope like this to mute warnings
export PYTHONWARNINGS="ignore::Warning"