Getting to Know HCL: Variables, Outputs, Locals, Expressions, count, for_each and Dynamic Blocks

Version note: The examples in this post were run with Terraform 1.17.2, AzureRM provider 5.14.0, the kreuzwerker/docker provider 4.7.0 and Docker Engine 29. For the Azure code, init, validate and fmt were run; no plan/apply was made against Azure.

Guess First

You create three Docker volumes for the Bookstore API with count: ["covers", "exports", "invoices"]. After a while exports is no longer needed and you delete it from the list. How many resources does the plan touch? Is only exports deleted, or is the invoices volume that holds the invoices affected too? The answer is in the middle of the post, with the real plan output.

Variables: Types, validation, sensitive and nullable

So far every value was written into the code: the network name, the port, the image tag. When we want to use the same code for dev, staging and production, these values have to come from outside. In Terraform the way to do that is input variables. The variables.tf file of the local example:

variable "environment" {
  type        = string
  description = "Deployment environment: dev, staging or prod."

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "The environment must be one of: dev, staging, prod."
  }
}

variable "api" {
  type = object({
    image_tag     = string
    external_port = number
    app_settings  = optional(map(string), {})
  })
  description = "Bookstore API container settings."

  validation {
    condition     = var.api.external_port >= 1024 && var.api.external_port <= 65535
    error_message = "The external_port must be between 1024 and 65535."
  }
}

variable "volume_names" {
  type        = set(string)
  description = "Data volumes mounted under /data in the API container."
  default     = ["covers", "exports"]
  nullable    = false
}

variable "payments_api_key" {
  type        = string
  description = "Key for the payment provider's sandbox."
  sensitive   = true
}

Every variable has a type. string, number and bool are simple types; list, set and map are collections of values of the same type; object is a structure where each field has its own type. The api variable is an object, and its app_settings field is defined as optional(map(string), {}): if it is not given, it becomes an empty map. It is similar to a property with a default value on a C# record.

validation blocks check the value before the plan. If condition does not return true, Terraform shows the error_message and stops. Giving the environment name as test:

Error: Invalid value for variable

  on variables.tf line 1:
   1: variable "environment" {
    ├────────────────
    │ var.environment is "test"

The environment must be one of: dev, staging, prod.

This was checked by the validation rule at variables.tf:5,3-13.

The same goes for the port: external_port = 80 gives "The external_port must be between 1024 and 65535." Validation catches a wrong value at the cheapest point, on your laptop, before it turns into a half-finished apply on Azure.

sensitive = true hides the value in plan and apply output. nullable = false makes Terraform use the default when someone deliberately passes null; when I tried -var 'volume_names=null', var.volume_names was still ["covers", "exports"]. That way you do not write a null check every time inside the module.

Supplying Values: tfvars Files and TF_VAR_

If you give no value to a variable that has no default, Terraform first asks interactively, and in a non-interactive environment like CI it stops:

Error: No value for required variable

  on variables.tf line 32:
  32: variable "payments_api_key" {

The root module input variable "payments_api_key" is not set, and has no
default value. Use a -var or -var-file command line argument to provide a
value for this variable.

Environment-specific values go in a .tfvars file. dev.tfvars:

environment = "dev"

api = {
  image_tag     = "1.1.0"
  external_port = 18480
  app_settings = {
    Logging__LogLevel__Default = "Information"
  }
}

This file is passed with terraform plan -var-file=dev.tfvars. If it were named terraform.tfvars or ended in *.auto.tfvars, Terraform would load it automatically; an explicit -var-file per environment causes fewer surprises. Secrets such as the payment provider's sandbox key do not go in tfvars, because the tfvars file is committed. They come from environment variables: every environment variable with the TF_VAR_ prefix sets the Terraform variable of the same name.

# PowerShell
$env:TF_VAR_payments_api_key = "<sandbox-key>"

# bash
export TF_VAR_payments_api_key="<sandbox-key>"

When a value comes from more than one place, the precedence is fixed: environment variables are lowest, then terraform.tfvars, then *.auto.tfvars files in alphabetical order, and at the top the -var and -var-file options on the command line, in the order they are written. That is why test wins when I run -var-file=dev.tfvars -var 'environment=test' in the validation example.

Locals and Expressions

A variable is a value that comes from outside; a local is an intermediate value computed inside the code. Instead of writing the same expression in five resources, you define it once as a local. The local example's locals.tf:

locals {
  name_prefix = "tfseries-${var.environment}"

  labels = {
    app         = "bookstore"
    environment = var.environment
    managed_by  = "terraform"
  }

  app_settings = merge(
    { Bookstore__Environment = var.environment },
    var.api.app_settings,
  )
}

name_prefix keeps the naming convention in one place: every Docker object will be named tfseries-dev-.... merge combines several maps; if a key appears more than once, the rightmost wins. So Bookstore__Environment is added in every environment automatically, and the app_settings from tfvars are layered on top.

The quickest way to try expressions and functions is terraform console. Running terraform console -var-file=dev.tfvars in the project folder opens a REPL where you can reach variables, locals and resources in the state:

> local.name_prefix
"tfseries-dev"
> merge(local.labels, { environment = "prod", owner = "platform" })
{
  "app" = "bookstore"
  "environment" = "prod"
  "managed_by" = "terraform"
  "owner" = "platform"
}
> lookup({ westeurope = "weu", northeurope = "neu" }, "swedencentral", "unknown")
"unknown"
> try(var.api.app_settings["Logging__LogLevel__Default"], "Warning")
"Information"
> try(var.api.app_settings["Missing__Key"], "Warning")
"Warning"
> [for name in var.volume_names : upper(name) if name != "exports"]
[
  "COVERS",
]
> convert({ port = "18480" }, object({ port = number }))
{
  "port" = 18480
}
> var.environment == "prod" ? "always" : "unless-stopped"
"unless-stopped"

lookup returns the third argument when the key is not in the map. try tries expressions in order and returns the result of the first one that does not fail; accessing a missing key normally fails, and try turns that into a default. A for expression corresponds to LINQ's Where and Select in C#: square brackets produce a list, curly braces a map, and if filters. The conditional expression (condition ? a : b) is the same as in C#.

convert arrived in Terraform 1.15 and explicitly converts a value to the type you write. Previously you used single-type functions like tonumber and tolist; convert also converts nested types such as objects in one go. For a value that cannot be converted (convert("abc", number)) it fails with "a number is required".

count or for_each?

There are two meta-arguments for several resources of the same type. I first create the volumes from the opening question with count:

resource "docker_volume" "data" {
  count = length(var.volume_names)
  name  = "tfseries-dev-${var.volume_names[count.index]}"
}

After the apply, the addresses in the state are docker_volume.data[0], [1] and [2]. Deleting exports from the list and running a plan:

  # docker_volume.data[1] must be replaced
-/+ resource "docker_volume" "data" {
      ~ driver      = "local" -> (known after apply)
      - driver_opts = {} -> null
      ~ id          = "tfseries-dev-exports" -> (known after apply)
      ~ mountpoint  = "/var/lib/docker/volumes/tfseries-dev-exports/_data" -> (known after apply)
      ~ name        = "tfseries-dev-exports" -> "tfseries-dev-invoices" # forces replacement
    }

  # docker_volume.data[2] will be destroyed
  # (because index [2] is out of range for count)
  - resource "docker_volume" "data" {
      - driver      = "local" -> null
      - driver_opts = {} -> null
      - id          = "tfseries-dev-invoices" -> null
      - mountpoint  = "/var/lib/docker/volumes/tfseries-dev-invoices/_data" -> null
      - name        = "tfseries-dev-invoices" -> null
    }

Plan: 1 to add, 0 to change, 2 to destroy.

The answer to the opening question: two resources are deleted and one is created. count identifies resources by their position. When the middle element goes, invoices moves from position 2 to position 1; Terraform reads that as "number 1 changed its name, number 2 no longer exists". As a result the invoices volume is deleted and created again empty; the invoices inside it are gone. Someone who applies this without reading the plan loses data they never touched.

Writing the same resource with for_each:

resource "docker_volume" "data" {
  for_each = toset(var.volume_names)
  name     = "tfseries-dev-${each.key}"
}

The addresses are now docker_volume.data["covers"], ["exports"], ["invoices"]. The plan for the same deletion:

  # docker_volume.data["exports"] will be destroyed
  # (because key ["exports"] is not in for_each map)
  - resource "docker_volume" "data" {
      - driver      = "local" -> null
      - driver_opts = {} -> null
      - id          = "tfseries-dev-exports" -> null
      - mountpoint  = "/var/lib/docker/volumes/tfseries-dev-exports/_data" -> null
      - name        = "tfseries-dev-exports" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Only the resource you wanted to delete is deleted. The rule is simple: if the resources are copies of each other and only the number matters (for example three identical workers), use count; if each one has an identity, use for_each. Where count is used most today is making a resource conditional: count = var.environment == "prod" ? 1 : 0. for_each accepts a map or a set of strings; if you pass a list you have to convert it with toset(). In the local example I defined the variable as set(string) from the start, which is why there is no toset in main.tf.

dynamic Blocks and Outputs

for_each produces resources; for repeated blocks inside a resource there is dynamic. In the Docker provider, labels and volume mounts are written as separate blocks. The local example's main.tf:

provider "docker" {}

resource "docker_network" "bookstore" {
  name = local.name_prefix

  dynamic "labels" {
    for_each = local.labels
    content {
      label = labels.key
      value = labels.value
    }
  }
}

resource "docker_volume" "data" {
  for_each = var.volume_names
  name     = "${local.name_prefix}-${each.key}"

  dynamic "labels" {
    for_each = local.labels
    content {
      label = labels.key
      value = labels.value
    }
  }
}

resource "docker_container" "api" {
  name    = "${local.name_prefix}-api"
  image   = "bookstore-api:${var.api.image_tag}"
  restart = var.environment == "prod" ? "always" : "unless-stopped"

  env = concat(
    [for key, value in local.app_settings : "${key}=${value}"],
    ["Bookstore__Payments__ApiKey=${var.payments_api_key}"],
  )

  dynamic "labels" {
    for_each = local.labels
    content {
      label = labels.key
      value = labels.value
    }
  }

  dynamic "volumes" {
    for_each = docker_volume.data
    content {
      volume_name    = volumes.value.name
      container_path = "/data/${volumes.key}"
    }
  }

  networks_advanced {
    name = docker_network.bookstore.name
  }

  ports {
    internal = 8080
    external = var.api.external_port
  }
}

dynamic "labels" produces one labels block for each key/value pair in the local.labels map; inside the block the iterator has the same name as the block (labels.key, labels.value). The container's dynamic "volumes" iterates directly over docker_volume.data: a resource created with for_each behaves like a map of its keys and instances. env, on the other hand, is a list attribute, not a block; we turn it into KEY=value strings with a for expression. The relevant part of the plan:

      + env                                         = (sensitive value)
...
      + labels {
          + label = "app"
          + value = "bookstore"
        }
      + labels {
          + label = "environment"
          + value = "dev"
        }
      + labels {
          + label = "managed_by"
          + value = "terraform"
        }
...
      + volumes {
          + container_path  = "/data/covers"
          + volume_name     = "tfseries-dev-covers"
            # (3 unchanged attributes hidden)
        }
      + volumes {
          + container_path  = "/data/exports"
          + volume_name     = "tfseries-dev-exports"
            # (3 unchanged attributes hidden)
        }

Look at the env line: because one of the list's elements comes from a sensitive variable, Terraform shows the whole list as (sensitive value). But that is only what is shown on screen. In the state file the value is plain text:

$ grep -c "Bookstore__Payments__ApiKey=" terraform.tfstate
1

Remember that sensitive is a display setting, not a security boundary; protecting the state is the subject of post 4, and never writing the secret to the state is the subject of post 9.

Outputs are the values the root module exposes. They are printed after an apply, can be read again with terraform output, and, as we will see in post 6, form half of the contract between modules:

output "api_url" {
  description = "Health endpoint of the local Bookstore API."
  value       = "http://localhost:${var.api.external_port}/health"
}

output "volume_names" {
  description = "Docker volume names, keyed by data set."
  value       = { for key, volume in docker_volume.data : key => volume.name }
}

output "app_settings" {
  description = "Non-secret settings passed to the API."
  value       = local.app_settings
}

After the apply:

$ terraform output
api_url = "http://localhost:18480/health"
app_settings = {
  "Bookstore__Environment" = "dev"
  "Logging__LogLevel__Default" = "Information"
}
volume_names = {
  "covers" = "tfseries-dev-covers"
  "exports" = "tfseries-dev-exports"
}
$ terraform output -json volume_names
{"covers":"tfseries-dev-covers","exports":"tfseries-dev-exports"}
$ curl http://localhost:18480/health
{"status":"Healthy","environment":"dev","version":"1.1.0"}

The -json output is for scripts and pipelines. If you mark an output that contains a secret with sensitive = true, terraform output shows it as <sensitive>, but -json and -raw print the value in the clear.

Bookstore's Azure Infrastructure in HCL

With the same tools we grow the infra/azure folder: a Linux App Service Plan, a Web App running a container, an Azure SQL Server with BookstoreDb, and a storage account with blob containers in it. Microsoft.Web and Microsoft.Sql were added to the registration list in the provider block. The variables look like the local example; the different ones are allowed_ip_ranges (the IP ranges allowed to reach the Web App), storage_containers, and the SQL administrator's login and password. The password is a sensitive variable and comes from TF_VAR_sql_admin_password. Just as in the local example, it is written to the state as plain text; in post 9 we close that with write-only arguments. The naming convention lives in locals:

locals {
  region_short = {
    westeurope    = "weu"
    northeurope   = "neu"
    swedencentral = "sdc"
  }

  # <type>-bookstore-<environment>-<region>, e.g. app-bookstore-dev-weu
  suffix         = "bookstore-${var.environment}-${lookup(local.region_short, var.location, var.location)}"
  compact_suffix = replace(local.suffix, "-", "")

  tags = {
    app         = "bookstore"
    environment = var.environment
    managed_by  = "terraform"
  }

  app_settings = merge(
    { Bookstore__Environment = var.environment },
    var.api.app_settings,
  )
}

Here lookup finds the region's short code and falls back to the region name itself. Because storage account names do not accept hyphens, replace also produces a version without them. The resources:

resource "azurerm_resource_group" "bookstore" {
  name     = "rg-${local.suffix}"
  location = var.location
  tags     = local.tags
}

resource "azurerm_storage_account" "bookstore" {
  name                     = "st${local.compact_suffix}"
  resource_group_name      = azurerm_resource_group.bookstore.name
  location                 = azurerm_resource_group.bookstore.location
  account_tier             = "Standard"
  account_replication_type = var.environment == "prod" ? "ZRS" : "LRS"
  min_tls_version          = "TLS1_2"
  tags                     = local.tags
}

resource "azurerm_storage_container" "bookstore" {
  for_each = var.storage_containers

  name                  = each.key
  storage_account_id    = azurerm_storage_account.bookstore.id
  container_access_type = "private"
}

resource "azurerm_service_plan" "bookstore" {
  name                = "asp-${local.suffix}"
  resource_group_name = azurerm_resource_group.bookstore.name
  location            = azurerm_resource_group.bookstore.location
  os_type             = "Linux"
  sku_name            = var.api.sku_name
  tags                = local.tags
}

resource "azurerm_linux_web_app" "api" {
  name                = "app-${local.suffix}"
  resource_group_name = azurerm_resource_group.bookstore.name
  location            = azurerm_resource_group.bookstore.location
  service_plan_id     = azurerm_service_plan.bookstore.id
  https_only          = true
  app_settings        = local.app_settings
  tags                = local.tags

  site_config {
    minimum_tls_version               = "1.2"
    health_check_path                 = "/health"
    health_check_eviction_time_in_min = 5
    ip_restriction_default_action     = length(var.allowed_ip_ranges) > 0 ? "Deny" : "Allow"

    application_stack {
      docker_image_name   = "${var.api.image_name}:${var.api.image_tag}"
      docker_registry_url = var.api.registry_url
    }

    dynamic "ip_restriction" {
      for_each = var.allowed_ip_ranges
      content {
        name       = ip_restriction.key
        ip_address = ip_restriction.value
        action     = "Allow"
        priority   = 100
      }
    }
  }
}

resource "azurerm_mssql_server" "bookstore" {
  name                         = "sql-${local.suffix}"
  resource_group_name          = azurerm_resource_group.bookstore.name
  location                     = azurerm_resource_group.bookstore.location
  version                      = "12.0"
  administrator_login          = var.sql_admin_login
  administrator_login_password = var.sql_admin_password
  minimum_tls_version          = "1.2"
  tags                         = local.tags
}

resource "azurerm_mssql_database" "bookstore" {
  name      = "BookstoreDb"
  server_id = azurerm_mssql_server.bookstore.id
  sku_name  = var.environment == "prod" ? "S1" : "Basic"
  tags      = local.tags
}

A few points to note. The blob containers are created with for_each; in AzureRM 5.x, azurerm_storage_container finds its storage account by storage_account_id, not by name. The dynamic "ip_restriction" block in the Web App's site_config produces one access rule per entry in the allowed_ip_ranges map; if the map is empty, no rule is created and the default action stays Allow. Conditionals carry the differences between environments: in production the storage uses ZRS and the database S1; in other environments LRS and Basic. The Web App's app_settings play the same role as the container env in the local example; ASP.NET Core reads Bookstore__Environment as Bookstore:Environment in configuration.

The outputs give the API's address, the SQL Server's host name and the container names:

output "api_url" {
  description = "Public URL of the Bookstore API."
  value       = "https://${azurerm_linux_web_app.api.default_hostname}"
}

output "sql_server_fqdn" {
  description = "Host name for the BookstoreDb connection string."
  value       = azurerm_mssql_server.bookstore.fully_qualified_domain_name
}

output "storage_containers" {
  description = "Blob container names, keyed by data set."
  value       = { for key, container in azurerm_storage_container.bookstore : key => container.name }
}

terraform validate says "Success! The configuration is valid." No plan was run against Azure; the pricing tiers of App Service and Azure SQL, SKU choice and region constraints belong to the Cloud Fundamentals series, not this one.

Production Notes

Design variables like a contract. Give every variable a description, keep its type as narrow as possible (an object with known fields instead of a loose map(any)), and guard the valid range of values with validation. Rejecting a wrong value before the plan is far cheaper than a half-finished apply on Azure.

Keep the naming convention in one place in locals and derive resource names from the environment. Many Azure resources cannot be renamed later; renaming means a replace. Getting the name right on day one is easier than migrating data afterwards.

Put secrets in environment variables, not tfvars, and make sure they do not show up in CI logs. sensitive hides them on screen, not in the state.

Common Pitfalls

Deleting the middle element of a list created with count. As you saw above, every following element shifts by one and is replaced. Use for_each for anything that has an identity. When converting an existing count resource to for_each, the addresses change too; doing that without deleting resources needs the moved blocks from post 8.

Giving for_each a value that is unknown at plan time. If the keys come from the ID of a resource that will be created during the apply, Terraform cannot know which instances to create and fails with "Invalid for_each argument". Keys should always be values known in the code or in variables.

Making everything a variable. Turning a value that never changes in any environment (for example account_tier = "Standard") into a variable does not make the code flexible; it only makes it harder to read. Values that really differ between environments become variables; the rest stays in the code.

A Small Experiment

★☆☆ Open terraform console -var-file=dev.tfvars, guess the result of { for k, v in local.labels : upper(k) => v }, then run it.

★★☆ Next to dev.tfvars, write a staging.tfvars (external_port = 18481, environment = "staging") and run terraform plan -var-file=staging.tfvars. Why does the plan want to change or recreate the existing dev resources? The answer to that question is the subject of post 7.

★★★ Add a third volume (invoices) to volume_names and run a plan. Why is the container replaced? Find the effect of the dynamic "volumes" block on the container in the plan output.

In the next post we get to the topic this post postponed several times: the state. What is in the state file, why it should not live on a laptop, how to move it to a remote backend, what locking does when two people apply at the same time, and when state mv, state rm and -refresh-only are needed.