Terraform: external yaml file as a contribution model for outside teams

If you are using Terraform as a way to provide infrastructure/services to outside teams, your Terraform definitions and variables may initially be owned by your team alone, with all “tf apply” operations done by trusted internal group members at the CLI.

But once there is a certain amount of maturity in the solution, the Terraform validate/plan/apply lifecycle is typically put into a CI/CD pipeline and driven by changes pushed to a git repository. And at that point, the external groups using your infra/services may want a more self-service approach.

So instead of logging work tickets requesting that your team “add one more X”, or “grant an additional role”, these external teams may simply want to make the proposed Terraform changes and log a git merge/pull request that your team can now approve and apply via the pipeline.

Changes driven by resources versus data

In general, changes to a Terraform plan can be user-initiated by:

Resource definition changes (manually changing ‘resource’, ‘data’, and ‘module’ declarations)
Data changes (manually changing simple variables and list/map)

Driven by resources

As a simple illustration, here is an example of manually defining the creation of two local files using hard-coded resources.

resource "local_file" "foo1" {
  content = "this is foo1"
  filename = "${path.module}/foo1.txt"
}

resource "local_file" "foo2" {
  content = "this is foo2"
  filename = "${path.module}/foo2.txt"
}

Driven by data

Versus the creation of two different files using data that drives a resource definition.

locals {
  file_data = [
    { name="foo1", content="this is foo1" },
    { name="foo2", content="this is foo2" }
  ]
}

resource "local_file" "foo" {
  for_each = { for entry in local.file_data : entry.name=>entry }
  content = each.value.content
  filename = "${path.module}/${each.value.name}.txt"
}

While it does take extra logic to parse the data structure, there is more flexibility in allowing data to control resources.

Contribution model driven by external yaml data source

Following the tenet of “changes driven by data”, it would be ideal if the changes submitted by external groups did not require any changes to our terraform .tf files. Because if you are looking at a merge/pull request coming from an external group, you really don’t want to worry about changes to the fundamental logic or the addition/removal of resources and modules, etc.

So that would lead toward all external changes being made in a variable file like “terraform.tfvars” or an “auto.tfvars“. This would be a step in the correct direction, but let me put forth the idea that an external yaml file that exposes only the variables used for external contributions is both easier to understand for the external teams and easier to review and approve for the controlling team.

As an example, let’s externalize the file names and content described earlier into their own “external.yaml” file.

$ cat external.yaml
files:
  - name: foo3
    content: |
      this is foo3
  - name: foo4
    content: |
      this is foo4

And now load that structure using yamldecode in our local variables section.

locals {
  file_data_ext_yaml = yamldecode(file("${path.module}/external.yaml"))
}

resource "local_file" "foo" {
  for_each = { for entry in local.file_data_ext_yaml.files : entry.name=>entry }
  content = each.value.content
  filename = "${path.module}/${each.value.name}.txt"
}

With this model, if external teams want to make a request such as “please add a file” or “change the content of this existing file”, they never need to touch our Terraform .tf files and instead can focus on modifications only to “external.yaml”.

This makes their life easier, because they never have to dig through terraform data declarations or for_each logic, they simply focus on the yaml data that defines the customization they desire. And it makes approving their merge/pull request easier, since the change has a known and limited scope.

Contribution model driven by external json data source

Of course, you could do something very similar with jsondecode and use an external json file, but I do think yaml has a simpler syntax if being edited by humans.

Ensure a base set of data, append with external data source

If you have a base set of data that must always be included, and you want external contributions to append to this base set, then use merge or concat (merge for maps, concat for lists).

Below is an example where we construct the local variable ‘file_data_merge’, which consists of a base set of data ‘file_data_static’ plus whatever is appended from the external yaml file.

locals {
  # base set of data
  file_data_static = [
    { name="foo1", content="this is foo1" },
    { name="foo2", content="this is foo2" }
  ]
  # external data
  file_data_ext_yaml = yamldecode(file("${path.module}/external.yaml"))

  # concat base and external data arrays
  file_data_merge = concat( local.file_data_static, local.file_data_ext_yaml.files )
}
resource "local_file" "foo" { 
  for_each = { for entry in local.file_data_merge : entry.name=>entry }
  content = each.value.content
  filename = "${path.module}/${each.value.name}.txt"
}