It would be uncommon to have one monolithic Terraform configuration for all the infrastructure in your organization. More than likely, there are multiple groups and each has responsibility and ownership of certain components (e.g. networking, storage, authorization, Kubernetes).
As an example, let’s say your responsibility is the Kubernetes cluster build. You may need the following values to complete this task:
- Network, subnet names, and CIDR blocks created by the Networking team
- Address and ports to shared storage buckets created by the Storage team
In its most rudimentary form, you could ask these teams to manually provide these values. But even better for automation and self-service, you could have terraform remote state pull the outputs from these teams’ terraform state, and use these values to populate your Cluster variables.
Overview
The terraform remote state data source allows you to pull in the state of another Terraform run, and use any of its exposed output variables in your own configuration. Notice this does not include any of the resources or data of the remote state, only root level output values.
This provides a mechanism for chaining independent Terraform runs, where the output of one configuration can populate the values of the next.
Terraform to create GCP project and bucket
As a prerequisites, you must have: a GCP account, and the terraform and gcloud utilities installed.
Then create a basic GCP project, with the state saved locally as ‘bootstrap.tfstate’.
# get project code git clone https://github.com/fabianlee/example-terraform_remote_state.git cd example-terraform_remote_state # gcloud login for terraform gcloud auth application-default login # sets env var 'TF_VAR_prject' with new random GCP project id source ./sourceMakeRandomProjectId.sh # creates basic project, saves to local bootstrap.tfstate cd gcp-project-bootstrap ./init.sh # shows output: bucketname, mybilling, projid, projname, projnumber, region ./apply.sh cd .. # validate local state and variable file ls -l bootstrap.tfstate cat dynamic-backend.hcl
The GCS storage bucket created inside the project will be the location of the remote state files in later runs.
Create GCP Topic, using local bootstrap state
Now let’s create a simple GCP Topic using Terraform. But instead of having to retype or assign values manually for basic values such as project and region, we will instead leverage the local bootstrap state.
Here is the snippet from gcp-topic/provider.tf:
# bootstrap state for base set of values
data "terraform_remote_state" "bootstrapstate" {
backend = "local"
config = {
path = "../bootstrap.tfstate"
}
}
provider "google" {
project = data.terraform_remote_state.bootstrapstate.outputs.projid
region = data.terraform_remote_state.bootstrapstate.outputs.region
}
Use gcloud to explicitly login to the new project first, then create this GCP Topic.
# gcloud login for terraform, set to explicit project id gcloud auth application-default login --project $TF_VAR_project # creates 'example-topic', saves state to remote GCS cd gcp-topic ./init.sh # shows output: topic1_labels, topic1_name ./apply.sh cd ..
Create another GCP Topic, using remote state of first Topic
Now let’s create another GCP Topic using Terraform. Once again we will load the local bootstrap state for the basic project values, but we will also load the remote state of the first Topic in order to gets its name and label values.
Here is the snippet from gcp-topic2/provider.tf:
# bootstrap state for base set of values
data "terraform_remote_state" "bootstrapstate" {
backend = "local"
config = {
path = "../bootstrap.tfstate"
}
}
# remote state from first Topic
data "terraform_remote_state" "topic1" {
backend = "gcs"
config = {
bucket = data.terraform_remote_state.bootstrapstate.outputs.bucketname
prefix = "gcp-topic"
}
}
And here is how the values are used to create the second Topic in gcp-topic2/main.tf:
resource "google_pubsub_topic" "example" {
# same name, but suffixed with '-2'
name = "${data.terraform_remote_state.topic1.outputs.topic1_name}-2"
# same labels, but add 'another'
labels = merge(data.terraform_remote_state.topic1.outputs.topic1_labels,{"another"="value"})
}
Then create this GCP Topic.
# creates 'example-topic-2', based on values from first Topic cd gcp-topic2 ./init.sh # shows output: topic1_name, topic2_name ./apply.sh cd ..
Validate GCP Infrastructure
After running the Terraform scripts above you should have two GCP topics, list them with the following gcloud command.
gcloud pubsub topics list --format='value(name,labels)'
Destroy Infrastructure
In order to avoid charges on the GCP infrastructure just created, you should tear down the GCP project.
cd gcp-project-bootstrap ./destroy.sh cd .. # remove local state and variables rm -f bootstrap.tfstate* dynamic-backend.hcl rm -fr gcp-project-bootstrap/.terraform gcp-project-bootstrap/.terraform.lock.hcl rm -fr gcp-topic/.terraform gcp-topic/.terraform.lock.hcl rm -fr gcp-topic2/.terraform gcp-topic2/.terraform.lock.hcl
REFERENCES
fabianlee, github project for this article with example usage of terraform_remote_state
hashicorp developer, using refresh-only to update remote state
hashicorp ref, terraform_remote_state
spacelift.io, terraform remote state data source
terraform.io, data source remote_state
terraform.io, resource google_storage_bucket
google ref, use terraform to create GCS bucket
ruanbekker, example amazon s3 remote state
hashicorp developer, using partial backend configuration for dynamic (since variable not possible)