Writing and Using Modules: Input/Output Contracts, Registry Modules, Versioning and Dynamic Sources

Version note: The examples in this post were run with Terraform 1.17.2, the kreuzwerker/docker provider 4.7.0, hashicorp/http 3.6.3, the Azure Verified Module avm-res-web-site 0.24.1 (AzAPI provider 2.13.0) and Docker Engine 29. For the AzureRM 5.14.0 code, init and validate were run; no plan/apply was made against Azure.

Guess First

In post 5 we put prevent_destroy = true on the API's data volumes. Now we move the container, the volumes and the image tracking into a module: we delete the resource definitions from the root, put them in the modules/bookstore-service folder and call the module from the root. The code is the same; only its location changed. What does the first plan say? Does prevent_destroy protect the volumes? The answer is in the first plan output.

When Do You Need a Module?

In Terraform every folder is a module. The folder where you run the terraform command is the root module; every folder the root calls with a module block is a child module. You can think of a child module as a function: its variables are the parameters, its outputs the return values, and the resources inside it the body.

There are two situations where a module really pays off. The first is when the same group of resources repeats in more than one place: if Bookstore's API, a future admin API and a background worker will all use the same "container + data volumes + labels" pattern, writing the pattern once and calling it three times keeps them consistent. The second is when one team wants to offer other teams a ready-made, vetted structure: like a platform team embedding the rule "every web app in the company is HTTPS-only, at least TLS 1.2 and has a managed identity" into a module.

There are also cases where it is unnecessary. A thin module that wraps a single resource with the same arguments (a wrapper around azurerm_storage_account that turns every argument into a variable) abstracts nothing; the reader now has to look at two files. If a pattern is used once, write it in the root first and extract it into a module when the second need arrives. Our reason for moving to a module in this post is the first one: we will reuse the API's structure for other services later.

The bookstore-service Module: Its Input/Output Contract

A module's outward face is its variables and outputs. The names, number or order of the resources inside can change; as long as the contract does not change, the caller is not affected. The relevant part of modules/bookstore-service/variables.tf:

variable "name" {
  type        = string
  description = "Container name."
}

variable "volume_prefix" {
  type        = string
  description = "Data volumes are named <volume_prefix>-<data set>; defaults to name."
  default     = null
}

variable "image" {
  type        = string
  description = "Image reference with a pinned tag, e.g. bookstore-api:1.1.0."

  validation {
    condition     = !endswith(var.image, ":latest")
    error_message = "Use a pinned image tag, not latest."
  }
}

variable "env" {
  type        = map(string)
  description = "Non-secret environment variables."
  default     = {}
  nullable    = false
}

variable "environment_variables" {
  type        = map(string)
  description = "Old name of env."
  default     = {}
  nullable    = false
  deprecated  = "Use env instead; environment_variables will be removed in the next major version of this module."
}

variable "secret_env" {
  type        = map(string)
  description = "Secret environment variables (still stored in the state)."
  default     = {}
  sensitive   = true
}

# networks, data_volumes, internal_port, external_port, restart, labels ...

There are a few design decisions. The module does not know the environment: it has no environment input. The "no latest in production" precondition from post 5 became a validation inside the module that applies to every environment: no container run with this module can use the latest tag. Environment-specific rules stay in the root. Secrets are in a separate secret_env input and are sensitive, so inside the module it is not lost which value is a secret. The resource blocks are the same as in post 5; only their addresses changed: docker_container.this, docker_volume.data, terraform_data.image. Calling the single resource of its kind in a module this is a common habit. The outputs:

output "container_name" {
  description = "Name of the running container."
  value       = docker_container.this.name
}

output "base_url" {
  description = "Base URL on the host."
  value       = "http://localhost:${var.external_port}"
}

output "volume_names" {
  description = "Volume names, keyed by data set."
  value       = { for key, volume in docker_volume.data : key => volume.name }
}

output "image_id" {
  description = "ID of the image the container runs."
  value       = data.docker_image.this.id
}

output "url" {
  description = "Old name of base_url."
  value       = "http://localhost:${var.external_port}"
  deprecated  = "Use base_url instead; url will be removed in the next major version of this module."
}

We will come back to the deprecated in the last output shortly. The module also has its own versions.tf:

terraform {
  required_version = ">= 1.15"

  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = ">= 4.0, < 5.0"
    }
  }
}

In the root we pinned the provider to a narrow range with ~> 4.7. The module has a wide range: "I work with 4.x". Every root that calls the module sets its own constraint, and Terraform uses the intersection of all of them. If a module sets a narrow constraint, it needlessly blocks provider upgrades in the roots that use it.

Calling the Module and Moving the Resources

We delete the container, volume and image resources from the root's main.tf and put the module call in their place:

module "api" {
  source = "../modules/bookstore-service"

  name          = "${local.name_prefix}-api"
  volume_prefix = local.name_prefix
  image         = "bookstore-api:${var.api.image_tag}"
  restart       = var.environment == "prod" ? "always" : "unless-stopped"
  env           = local.app_settings
  secret_env    = { Bookstore__Payments__ApiKey = var.payments_api_key }
  networks      = [docker_network.bookstore.name, data.docker_network.shared.name]
  data_volumes  = var.volume_names
  external_port = var.api.external_port
  labels        = local.labels
}

The volume_names output now returns module.api.volume_names, and the check block takes its URL from module.api.base_url. When a module is added, terraform init runs again:

Initializing modules...
- api in ../modules/bookstore-service

The answer to the opening question is in the first plan:

  # docker_container.api will be destroyed
  # (because docker_container.api is not in configuration)
  # docker_volume.api_data["covers"] will be destroyed
  # (because docker_volume.api_data is not in configuration)
  # docker_volume.api_data["exports"] will be destroyed
  # (because docker_volume.api_data is not in configuration)
  # terraform_data.api_image will be destroyed
  # (because terraform_data.api_image is not in configuration)
  # module.api.docker_container.this will be created
  # module.api.docker_volume.data["covers"] will be created
  # module.api.docker_volume.data["exports"] will be created
  # module.api.terraform_data.image will be created
Plan: 4 to add, 0 to change, 4 to destroy.

To Terraform, docker_volume.api_data and module.api.docker_volume.data are two different addresses; it sees one leaving the code and the other arriving. The volumes are not protected, because the limit we saw in post 5 applies here: prevent_destroy works only while the resource block is in the code. When the block left the code, the protection left with it. Had this plan been applied, the cover images and the exports would have been deleted.

The way to tell Terraform about the move is moved blocks. They are the in-code, pull-request-reviewed equivalent of state mv from post 4:

# The API resources moved into the bookstore-service module (post 8 covers moved blocks in detail)
moved {
  from = docker_container.api
  to   = module.api.docker_container.this
}

moved {
  from = docker_volume.api_data
  to   = module.api.docker_volume.data
}

moved {
  from = terraform_data.api_image
  to   = module.api.terraform_data.image
}

On my first attempt the plan still wanted to delete something, and this time prevent_destroy stepped in:

  # module.api.docker_volume.data["covers"] must be replaced
  # (moved from docker_volume.api_data["covers"])
...
      ~ name        = "tfseries-dev-covers" -> "tfseries-dev-api-covers" # forces replacement
...
Error: Instance cannot be destroyed

  on ../modules/bookstore-service/main.tf line 17:
  17: resource "docker_volume" "data" {

Resource module.api.docker_volume.data["covers"] has
lifecycle.prevent_destroy set, but the plan calls for this resource to be
destroyed. To avoid this error and continue with the plan, either disable
lifecycle.prevent_destroy or reduce the scope of the plan using the -target
option.

The first version of the module derived the volume names from the container name (tfseries-dev-api-covers), while the old code used the environment prefix (tfseries-dev-covers). Moving the address was not enough; because the name changed, a replace was needed. Naming is part of a module's contract. After adding a volume_prefix input whose default is the container name and passing local.name_prefix from the root, the plan came out clean:

  # docker_container.api has moved to module.api.docker_container.this
    resource "docker_container" "this" {
        id                                          = "1284850060aeaccace28350326fd4c6844eca43127f96f05cacadc66452cf462"
        name                                        = "tfseries-dev-api"
        # (51 unchanged attributes hidden)

        # (8 unchanged blocks hidden)
    }
...
  # docker_volume.api_data["covers"] has moved to module.api.docker_volume.data["covers"]
  # docker_volume.api_data["exports"] has moved to module.api.docker_volume.data["exports"]
  # terraform_data.api_image has moved to module.api.terraform_data.image
...
Plan: 0 to add, 0 to change, 0 to destroy.

After the apply, the addresses in the state are inside the module:

$ terraform state list
data.docker_network.shared
data.http.health
docker_network.bookstore
module.api.data.docker_image.this
module.api.docker_container.this
module.api.docker_volume.data["covers"]
module.api.docker_volume.data["exports"]
module.api.terraform_data.image

The container was never recreated. Do not delete the moved blocks right away; they need to stay in the code until everyone on the team and every environment has applied this change. We will discuss when to delete them in post 8.

The Azure Side: The bookstore-webapp Module

We make the same split in the Azure code. The App Service Plan, the managed identity, the Key Vault role assignment and the Web App moved to modules/bookstore-webapp; the resources inside were named this, and depends_on and ignore_changes went with them. The call in the root:

module "api" {
  source = "../modules/bookstore-webapp"

  name_suffix         = local.suffix
  resource_group_name = azurerm_resource_group.bookstore.name
  location            = azurerm_resource_group.bookstore.location
  sku_name            = var.api.sku_name
  image               = { name = var.api.image_name, tag = var.api.image_tag, registry_url = var.api.registry_url }
  allowed_ip_ranges   = var.allowed_ip_ranges
  key_vault           = { id = data.azurerm_key_vault.shared.id, name = data.azurerm_key_vault.shared.name }
  tags                = local.tags

  app_settings = local.app_settings
}

The module's contract is this: a name suffix, the resource group and region, the SKU, the initial image, app settings, the IP ranges allowed in, and the ID and name of the Key Vault to read. In return it gives the URL, the Web App ID and the identity's principal ID; the last one lets the root grant other roles to that identity. In the root, moved.tf moves the four resources to their new addresses, and the check block takes its URL from module.api.url. I did not run this code on Azure; init found the module and validate passed.

One thing changed places: the "no B1 plan in production" precondition from post 5 could not go into the module, because it needs to know the environment. A module block has no lifecycle, so the rule moved to the validation of the api variable in the root; since Terraform 1.9, validations can also look at other variables.

Module Sources and Version Pinning

The source argument says where a module comes from. Three common sources:

# A folder in the same repository: the version is the repository's commit
source = "../modules/bookstore-service"

# Another git repository: pinned to a tag with ref
source = "git::https://github.com/example/bookstore-terraform-modules.git//bookstore-webapp?ref=v1.4.0"

# Terraform Registry: a semver constraint with version
source  = "Azure/avm-res-web-site/azurerm"
version = "0.24.1"

A local path has no separate version; the module changes in the same commit as the code that calls it. That is the simplest option for a team living in a single repository. If modules live in a separate repository and are used by more than one team, you need a git source pinned to a tag: a git source without ?ref= pulls the default branch, and the moment the module's owner pushes a change, your next init downloads different code. The repository in the git example is only there to show the format; I did not run it. The // marks a subfolder inside the repository.

Modules in the Terraform Registry take a semver constraint with version. Microsoft's Azure Verified Modules (AVM) project publishes Microsoft-supported modules for Azure resources there. I called the Web App module in a separate example folder:

# Example only: the same web app from an Azure Verified Module instead of our own module
terraform {
  required_version = "~> 1.17"

  required_providers {
    azapi = {
      source  = "Azure/azapi"
      version = "~> 2.12"
    }
  }
}

provider "azapi" {}

module "web" {
  source  = "Azure/avm-res-web-site/azurerm"
  version = "0.24.1"

  name                     = "app-bookstore-dev-weu"
  location                 = "westeurope"
  parent_id                = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-bookstore-dev-weu"
  service_plan_resource_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-bookstore-dev-weu/providers/Microsoft.Web/serverFarms/asp-bookstore-dev-weu"
  enable_telemetry         = false
}

Initializing modules...
Downloading registry.terraform.io/Azure/avm-res-web-site/azurerm 0.24.1 for web...
- web in .terraform/modules/web
- web.config_slotconfignames in .terraform/modules/web/modules/config_slotconfignames
...
Downloading registry.terraform.io/Azure/avm-utl-interfaces/azure 0.7.0 for web.avm_interfaces...
...
$ terraform providers
Providers required by configuration:
.
├── provider[registry.terraform.io/azure/azapi] ~> 2.12
└── module.web
    ├── provider[registry.terraform.io/hashicorp/time] >= 0.9.0, < 1.0.0
    ├── provider[registry.terraform.io/azure/azapi] ~> 2.12
    ├── provider[registry.terraform.io/azure/modtm] ~> 0.3
    ├── provider[registry.terraform.io/hashicorp/random] >= 3.5.0, < 4.0.0
...

This output teaches us three things. First, despite having azurerm in its name, the current version of the module does not use the AzureRM provider but the AzAPI provider, which calls Azure's REST API directly (2.13.0 went into the lock file); we are not on the same provider as our own code. Second, for a small web app, 329 .tf files and a 7.5 MB module tree were downloaded: that is the price of a module covering every App Service feature. Third, modtm is AVM's telemetry provider; the module's enable_telemetry input defaults to true, and I turned it off. validate passed.

As for version pinning: in a module below 1.0, semver allows even a minor version to bring breaking changes, so pin 0.x modules like AVM to an exact version ("0.24.1") and upgrade after reading the changelog. For modules at 1.0 and above, a constraint like ~> 1.4 is enough. Either way, read the first plan after an upgrade carefully: a changed default inside a module can produce a replace without a single line of your code changing.

My recommendation for Bookstore: our own small modules in the same repository, referenced by local path. Registry modules like AVM only when maintaining our own code would cost more than maintaining theirs.

Terraform 1.15: Variables in Module Sources and deprecated

Until Terraform 1.15, source and version only accepted literal values, because modules are downloaded during init, before variables are evaluated. Since 1.15 they accept locals and variables marked const = true. When I tried using a variable without const in the source:

Error: Unknown module source

  on main.tf line 27, in module "api":
  27:   source = "${var.modules_path}/bookstore-service"

Only literal values and const variables can be evaluated during init.

With the variable marked const:

variable "modules_path" {
  type        = string
  description = "Where shared modules live; a git URL with ?ref= in CI."
  default     = "../modules"
  const       = true
}

$ terraform init -var 'modules_path=../../lab/modules'
...
- api in ../../lab/modules/bookstore-service
Terraform has been successfully initialized!

I ran this experiment on a copy of the code; in Bookstore's real code the module path stays literal. The real use case is managing the same module version for every environment folder from one place (for example a ref or version value in a local); we will use it in post 7 when we split environments. A const variable cannot depend on a value computed during the plan; it only takes constant values.

The second feature 1.15 brought is deprecated for variables and outputs. The hardest part of modules is changing the contract: if you rename an input, every root that uses it breaks on its next init. In the first version of bookstore-service the input was called environment_variables; later it became env. Instead of deleting the old name, we marked it with deprecated (the variables.tf above). When the root uses the old name, the plan keeps working but warns:

Warning: Deprecated variable got a value

  on main.tf line 33, in module "api":
  33:   environment_variables = local.app_settings

Use env instead; environment_variables will be removed in the next major
version of this module.

The same goes for outputs: the url output became base_url, and the old one is deprecated. Using module.api.url in the check block:

Warning: Deprecated value used

  with data.http.health,
  on main.tf line 43, in check "api_health":
  43:     url = "${module.api.url}/health"

  The deprecation originates from module.api.url

Use base_url instead; url will be removed in the next major version of this
module.

According to the documentation, a deprecated output can only be used in child modules, not in the root module. The flow is: add the new name, mark the old one with deprecated, give the module's users one release of transition time, and delete the old one in the next major version.

Production Notes

Run a module like an API that has users. The descriptions of its variables and outputs are its documentation; renaming them is a breaking change. For modules living in a separate repository, use semver, write a changelog line for every release, and mark inputs that will be removed with deprecated at least one release in advance.

Before moving resources into a module, always run the plan together with the moved blocks and see 0 to destroy on the last line. The move is a refactor; nothing in the infrastructure should change. If something does, the reason is a difference you missed in the contract, like our volume names.

Review the code and providers a registry module pulls in. A 329-file module is a dependency that will run in your infrastructure and change with every upgrade. Set defaults such as telemetry deliberately.

Common Pitfalls

Wrapping every resource in a module. Single-resource modules such as a "storage account module" or a "resource group module" abstract nothing; they only add a layer of indirection. A module is written for a group of resources that means something together.

Defining a provider configuration inside a module. The root that calls the module should pass its provider down; if the module has its own provider block, you cannot call it with for_each. When I added provider "docker" to a copy of bookstore-service and called it with for_each for two environments:

Error: Module is incompatible with count, for_each, and depends_on

  on main.tf line 3, in module "api":
   3:   for_each = toset(["dev", "staging"])

The module at module.api is a legacy module which contains its own local
provider configurations, and so calls to it may not use the count, for_each,
or depends_on arguments.

A module contains only required_providers; the provider itself, its credentials and the subscription ID are defined in the root.

Leaving a git source without a ref, or pinning it to a branch name. ?ref=main is not pinning; the moment the module's owner pushes to main, your infrastructure plans with different code. Use a tag or a commit SHA.

A Small Experiment

★☆☆ Compare the output of terraform state list before and after moving to the module. Which addresses changed and which stayed the same?

★★☆ Call bookstore-service a second time to add a container for the future admin API (with a different host port). How many lines did you have to write in the root? Did a difference between the two containers come up that the module does not cover?

★★★ Delete the module's url output entirely and leave a line in the root that still uses module.api.url. In which command does the error appear, and with which message? This shows why the transition time you give with deprecated matters.

In the next post we split environments: CLI workspaces, a directory per environment, or a single codebase with tfvars per environment? Splitting the state into layers, blast radius, and the recommended folder layout for Bookstore.