Terraform: importing a module by its individual resources


Categories: DevOps

If you need to “import a Terraform module”, it is critical to understand that importing a module state is not a single bundled operation.  Instead, you must import each of the resources inside the module individually.

It is unfortunate that you cannot simply supply the module identifier and its variable values to import all its resources in a single operation, but that is not the way it is implemented.

Simple module example

I have created an example project on github named tf-module-for-import.  In main.tf, it includes a module from a child sub-directory that takes a single “name” argument.

module "mysimplemod" {
  source = "./gcp-storage-bucket"
  name = "test"
}

This module creates a GCP storage bucket and service account, but please ignore the hyperscaler specifics because this article really focuses on how module import is accomplished in general.

Terraform module import concept

It would be tempting to believe “importing a terraform module” might conceptually be a single call like below.  Especially given that removing a module’s state has this model.

# this is indeed the correct way to remove module state
terraform state rm module.mysimplemod

# but this is NOT the model for import !!!!
terraform import module.mysimplemod --var name=test

But this is NOT the model for importing a module’s state.  Instead, you must individually import each resource inside the module.

If we looked inside our example “gcp-storage-bucket” module directory, we would see the individual resources for the module.  Below is a snippet of the resources defined in our example module.

resource "random_id" "bucket_prefix" {
  byte_length = 8
}

resource "google_service_account" "default" {
  account_id   = "storage-sa-${var.name}"
}

resource "google_storage_bucket" "default" {
  name     = "bucket-${var.name}-${random_id.bucket_prefix.hex}"
  location = "US"
}

In order to import the state of the module, we need to execute an import operation on each of these resources individually.

#terraform import <ADDR> <ID>
terraform import module.mysimplemod.random_id.bucket_prefix ${THE_RAND_ID}
terraform import module.mysimplemod.google_service_account.default projects/${GOOGLE_PROJECT}/serviceAccounts/storage-sa-${THE_NAME}@${GOOGLE_PROJECT}.iam.gserviceaccount.com
terraform import module.mysimplemod.google_storage_bucket.default projects/bucket-${THE_NAME}-${THE_RAND_HEX}

The three commands above looks reasonable, but there are definitely challenges with putting together this list for real-world modules.  Even looking at the commands above, you can see that although the module address (<ADDR>) is relatively straight forward, the syntax required for the resource id (<ID>) is very specific and often requires that you understand the full context of its provisioning.

Terraform module import challenges

There are multiple challenges you will face when putting together this list of “terraform import” commands for a module.

  • Manual toil of identifying each resource name, there can be a considerable number
  • Resources based on list/map with for_each that requires you manually “unroll the loop” and import each resource index
  • Nested modules within modules that require you dive deeper into the resources graph, hence more import commands
  • Finding remote module source code, so you can identify the entire list of resources that need to be imported
  • Identifying the exact module version in use locally, so you are not looking at older/newer resource code definitions
  • Looking up the documentation for every single resource to find the exact identifier syntax it requires
  • Not every terraform resource supports import

There is not a simple solution to these issues, it requires a lot of work looking up documentation and source code, then iterative testing to make sure all the resources are recognized.

Here are some tips that may assist you during this work:

  • Use ‘terraform plan’ to see the <ADDR> that terraform believes are missing from state
  • If you are having trouble with the syntax for a complex resource <ID>, try looking inside the json terraform.tfstate file at similar resources
  • Import does not always bring in every attribute required to get a clean “plan/apply”, you can use lifecycle.ignore_changes if you must avoid action on a resource
  • terraform.tfstate is just a json file, if you must manually insert a json resource or tweak a string attribute, that is a possibility you can consider
  • Use ‘terraform state list’ to see what is currently contained within the state file

 

REFERENCES

Terraform modules

gthub tf-modules-for-import, example code for this article

Terraform docs, lifecycle and ignore_changes

Yakuphan, article about modules and import

Jack Roper, how to use ignore_changes in terraform lifecycle

Flavius Dinu,  importing existing infrastructure into Terraform