Validating and Testing Terraform Code: fmt, validate, tflint, terraform test, mock_provider and Policy

Version note: The examples in this post were run with Terraform 1.17.2, TFLint 0.66.0 (tflint-ruleset-azurerm 0.33.0), Trivy 0.76.2, the kreuzwerker/docker provider 4.7.0 and Docker Engine 29. The AzureRM 5.14.0 code was tested with terraform test using mock_provider; no plan/apply was made against Azure.

Guess First

In the bookstore-webapp module of Bookstore (the online bookstore used as the shared example across this site's series), someone lowers minimum_tls_version from "1.2" to "1.1"; apparently an old client could not connect. Does terraform fmt -check pass? Does terraform validate pass? If neither catches it, what can you write, without an Azure account, that stops this change in the pull request?

Layers: What Does Each Check Catch?

The answer to the first two questions: both pass. fmt only looks at formatting. validate checks syntax, references, types and the provider schema; "1.1" is one of the values AzureRM accepts, so it is valid. What is wrong is not the code but our rule. Validating Terraform code is a job of several layers, and each layer catches what the previous one cannot see:

We saw an example of this in post 7: the validation that forbids B1 in the prod folder ran during plan, not during validate. The way to get a plan without going to Azure is mock_provider.

tflint: What validate Cannot See

tflint is a separate tool that reads Terraform code like a linter. The terraform ruleset that comes with it finds unused variables, variables without types, providers without pinned versions and similar problems. Provider-specific rulesets can be added; the AzureRM one checks invalid VM sizes, missing settings and some good-practice rules. infra/.tflint.hcl:

plugin "terraform" {
  enabled = true
  preset  = "recommended"
}

plugin "azurerm" {
  enabled = true
  version = "0.33.0"
  source  = "github.com/terraform-linters/tflint-ruleset-azurerm"
}

# The health check (health_check_path) covers this for Bookstore
rule "azurerm_app_service_missing_auto_heal_setting" {
  enabled = false
}

I added the rule block at the end of the file after the first run. The first result:

$ cd infra
$ tflint --init
Installing "azurerm" plugin...
Installed "azurerm" (source: github.com/terraform-linters/tflint-ruleset-azurerm, version: 0.33.0)
$ tflint --recursive --config "$(pwd)/.tflint.hcl" --format compact
8 issue(s) found:

modules/bookstore-webapp/main.tf:23:1: Warning - auto_heal_setting should be configured in site_config block for robust app services. (azurerm_app_service_missing_auto_heal_setting)
bootstrap-azure/main.tf:36:1: Warning - Resource is missing lifecycle { prevent_destroy = true }. This resource contains data that should be protected from accidental deletion. (azurerm_resources_missing_prevent_destroy)
bootstrap-azure/main.tf:60:1: Warning - Resource is missing lifecycle { prevent_destroy = true }. ...
azure/bookstore/secrets.tf:13:1: Warning - Resource is missing lifecycle { prevent_destroy = true }. ...
azure/bookstore/main.tf:45:1: Warning - Resource is missing lifecycle { prevent_destroy = true }. ...
azure/bookstore/main.tf:7:1: Warning - Resource is missing lifecycle { prevent_destroy = true }. ...
azure/bookstore/main.tf:17:1: Warning - Resource is missing lifecycle { prevent_destroy = true }. ...
azure/bookstore/main.tf:17:1: Warning - Resource is missing lifecycle { prevent_destroy = true }. ...

The terraform ruleset found nothing. The AzureRM ruleset said two things. First, seven data-carrying resources have no prevent_destroy: Bookstore's storage account and blob containers, SQL Server, the Key Vault secret, and the storage account and container that hold the state. We put it on BookstoreDb in post 5 and skipped the rest. I added it to the storage, SQL Server and state resources. For the Key Vault secret I turned the rule off on that resource only, because rotation does not delete the secret, it adds a new version:

# Generated in memory on every run; Terraform never writes it to state or plan
ephemeral "random_password" "sql_admin" {
  length           = 32
  special          = true
  override_special = "!#%*-_=+"
  min_lower        = 2
  min_upper        = 2
  min_numeric      = 2
  min_special      = 2
}

# Break-glass copy for people; the app connects with its managed identity, not this password
# tflint-ignore: azurerm_resources_missing_prevent_destroy # rotation replaces the value, not the secret
resource "azurerm_key_vault_secret" "sql_admin_password" {
  name             = "bookstore-sql-admin-password"
  key_vault_id     = data.azurerm_key_vault.shared.id
  content_type     = "password"
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = var.sql_admin_password_version
}

Second, the Web App has no auto_heal_setting. I turned this rule off in the config; in Bookstore the health_check_path setting takes unhealthy instances out of rotation. Turning a rule off is a legitimate decision, as long as the reason is written in the code. After the fixes tflint comes back clean.

terraform test and mock_provider

terraform test is the test framework that has shipped with Terraform since 1.6. Test files have the .tftest.hcl extension and usually live in the module's tests folder. Each file consists of one or more run blocks; each run executes a plan or an apply and checks the result with assert blocks. mock_provider, introduced in Terraform 1.7, stands in for the provider and produces fake values for every resource; it never goes to the cloud. The test that stops the change from the guess:

# Runs without Azure: every azurerm resource and data source gets fake values
mock_provider "azurerm" {}

variables {
  name_suffix         = "bookstore-test-weu"
  resource_group_name = "rg-bookstore-test-weu"
  location            = "westeurope"
  image               = { name = "bookstore/bookstore-api", tag = "1.1.0" }
  key_vault = {
    id   = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-platform-test-weu/providers/Microsoft.KeyVault/vaults/kv-platform-test-weu"
    name = "kv-platform-test-weu"
  }
}

run "https_and_tls" {
  command = plan

  assert {
    condition     = azurerm_linux_web_app.this.https_only == true
    error_message = "The web app must accept HTTPS only."
  }

  assert {
    condition     = azurerm_linux_web_app.this.site_config[0].minimum_tls_version == "1.2"
    error_message = "The minimum TLS version must be 1.2."
  }
}

run "open_without_ip_ranges" {
  command = plan

  assert {
    condition     = azurerm_linux_web_app.this.site_config[0].ip_restriction_default_action == "Allow"
    error_message = "Without IP ranges the app must stay reachable."
  }
}

run "deny_by_default_with_ip_ranges" {
  command = plan

  variables {
    allowed_ip_ranges = { office = "203.0.113.0/24" }
  }

  assert {
    condition     = azurerm_linux_web_app.this.site_config[0].ip_restriction_default_action == "Deny"
    error_message = "With IP ranges, everything else must be denied."
  }

  assert {
    condition     = length(azurerm_linux_web_app.this.site_config[0].ip_restriction) == 1
    error_message = "Expected one IP restriction rule."
  }
}

The variables block gives values shared by all run blocks; the last run adds its own. The assertions look at the planned values of the resource inside the module. In the module folder:

$ cd infra/modules/bookstore-webapp
$ terraform test
tests/webapp.tftest.hcl... in progress
  run "https_and_tls"... pass
  run "open_without_ip_ranges"... pass
  run "deny_by_default_with_ip_ranges"... pass
tests/webapp.tftest.hcl... tearing down
tests/webapp.tftest.hcl... pass

Success! 3 passed, 0 failed.

The whole file finished in 1.6 seconds; no Azure account or internet connection was needed. Now I lower the TLS version to "1.1":

$ terraform test
tests/webapp.tftest.hcl... in progress
  run "https_and_tls"... fail

Error: Test assertion failed

  on tests/webapp.tftest.hcl line 24, in run "https_and_tls":
  24:     condition     = azurerm_linux_web_app.this.site_config[0].minimum_tls_version == "1.2"
    ├────────────────
    │ Diff:
    │ --- actual
    │ +++ expected
    │ - "1.1"
    │ + "1.2"


The minimum TLS version must be 1.2.
  run "open_without_ip_ranges"... pass
  run "deny_by_default_with_ip_ranges"... pass
tests/webapp.tftest.hcl... tearing down
tests/webapp.tftest.hcl... fail

Failure! 2 passed, 1 failed.

The test shows what the value should be and what it is. The change that validate let through stops here. With 1.17, mock_provider also supports ephemeral resources; that is what makes the code with the ephemeral random_password from post 9 testable.

"Applying" a Whole Environment Against Mocks

command = plan is fast, but some values are unknown at plan time. To test the prod rule from post 7 and the environment differences on the whole infra/azure/bookstore module, I ran one run with command = apply, still against mocks. On the first try:

$ terraform test
...
Error: parsing "kkrxyvy7": parsing the StorageAccount ID: the number of segments didn't match
...
  with azurerm_storage_container.bookstore["covers"],
...
Error: parsing "obnpp4zv": parsing the SqlServer ID: the number of segments didn't match
...
Error: expected "principal_id" to be a valid UUID, got u4tk8v3c

  with module.api.azurerm_role_assignment.key_vault_secrets,
...
Failure! 0 passed, 1 failed, 2 skipped.

The mock provider produces fake values as random strings, but the provider's own validations still run: the blob container expects the storage account ID in Azure format, and the role assignment expects a UUID principal ID. So for types whose IDs are used by another resource, you need realistic defaults:

mock_provider "azurerm" {
  # ...

  # The provider still validates IDs during apply, so random mock strings are not enough
  mock_resource "azurerm_storage_account" {
    defaults = {
      id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-bookstore-dev-weu/providers/Microsoft.Storage/storageAccounts/stbookstoredevweu"
    }
  }

  mock_resource "azurerm_mssql_server" {
    defaults = {
      id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-bookstore-dev-weu/providers/Microsoft.Sql/servers/sql-bookstore-dev-weu"
    }
  }

  # ... azurerm_service_plan, azurerm_user_assigned_identity
}

The rest of the file gives the Key Vault data source an ID in the same format, mocks the http data source in the check block to return 200, and defines three run blocks: dev's small tiers, rejecting B1 in prod (expect_failures = [var.api]), and zone-redundant storage with an S1 database in prod. The result:

$ cd infra/azure/bookstore
$ terraform test
tests/bookstore.tftest.hcl... in progress
  run "dev_uses_small_tiers"... pass
  run "prod_rejects_b1"... pass
  run "prod_uses_zone_redundancy"... pass
tests/bookstore.tftest.hcl... tearing down
tests/bookstore.tftest.hcl... pass

Success! 3 passed, 0 failed.

What is tested here is not Azure's behaviour but which values our code produces. A mocked test does not catch whether Azure offers a SKU in that region, name clashes or missing permissions. Those need tests that run with the real provider in a real test subscription.

Testing with the Real Provider: bookstore-service

In the local example, testing with the real provider is cheap because we have Docker. The test for the bookstore-service module creates a container and a volume and checks them:

# Uses the real Docker provider: the apply run creates and removes a container
provider "docker" {}

variables {
  name          = "tfseries-test-api"
  image         = "bookstore-api:1.1.0"
  networks      = ["tfseries-shared"]
  data_volumes  = ["covers"]
  external_port = 18482
}

run "rejects_latest_tag" {
  command = plan

  variables {
    image = "bookstore-api:latest"
  }

  expect_failures = [var.image]
}

run "starts_container_with_volume" {
  assert {
    condition     = output.volume_names["covers"] == "tfseries-test-api-covers"
    error_message = "Volume names must be <name>-<data set>."
  }

  assert {
    condition     = docker_container.this.restart == "unless-stopped"
    error_message = "The default restart policy must be unless-stopped."
  }
}

$ cd infra/modules/bookstore-service
$ terraform test
tests/service.tftest.hcl... in progress
  run "rejects_latest_tag"... pass
  run "starts_container_with_volume"... pass
tests/service.tftest.hcl... tearing down
tests/service.tftest.hcl... pass

Success! 2 passed, 0 failed.
$ docker volume ls --filter name=tfseries-test
DRIVER    VOLUME NAME

The first run confirms with expect_failures that the "no latest tag" validation from post 6 really works: when the error appears where expected, the test passes. The second run has no command, so it defaults to apply: the container was really created, the assertions looked at real values, and when the test finished Terraform deleted everything. One detail caught my eye: the module's volumes have prevent_destroy, yet the test teardown deleted the volume. This is not a bug; the Terraform 1.14 changelog states that prevent_destroy is ignored during test cleanup. It makes sense for the test framework to be able to clean up what it created, but it also shows why running tests against a real environment's state is dangerous.

Security Scanning: Trivy, Checkov and Policy

Security scanners check resource settings against known rule sets: public access to storage, encryption, logging, TLS. I ran Trivy over the whole infra folder:

$ trivy config --disable-telemetry infra
...
┌──────────────────────────────────┬───────────┬───────────────────┐
│              Target              │   Type    │ Misconfigurations │
├──────────────────────────────────┼───────────┼───────────────────┤
│ azure/bookstore/main.tf          │ terraform │        21         │
├──────────────────────────────────┼───────────┼───────────────────┤
│ azure/bookstore/secrets.tf       │ terraform │         3         │
├──────────────────────────────────┼───────────┼───────────────────┤
│ bootstrap-azure/main.tf          │ terraform │         5         │
├──────────────────────────────────┼───────────┼───────────────────┤
│ modules/bookstore-webapp/main.tf │ terraform │         9         │
└──────────────────────────────────┴───────────┴───────────────────┘
...
AZU-0012 (CRITICAL): No network rules defined and default action allows access.
════════════════════════════════════════
The default_action for network rules should come into effect when no other rules are matched.
The default action should be set to Deny.
See https://avd.aquasec.com/misconfig/azu-0012
────────────────────────────────────────
 azure/bookstore/main.tf:7-19
   via azure/envs/dev/main.tf:1-19 (module.bookstore)

There were 38 findings; some are the same rule repeated in three environments, and the number of distinct rules is 11. Instead of treating them all with the same weight, I split them into three groups. To fix right away: HTTP/2 being off on App Service is a one-line job, so I added http2_enabled = true. Needing an architecture decision: the storage account and SQL Server being open to the public network (AZU-0012 and AZU-0022). The right fix is private endpoints and a VNet; that is the subject of the Cloud Fundamentals series and will be closed once Bookstore's network layer is in place. Accepted on purpose: rules whose cost or complexity exceeds what Bookstore needs, such as customer-managed key encryption, client certificates and geo-redundant replication. Keeping accepted findings in a .trivyignore file with their reasons stops the report from showing the same list again on the next scan.

Checkov does the same job with a Python-based and broader rule set; OPA/Conftest lets you write the rules yourself in the Rego language, for company-specific rules such as "every resource must have app and environment tags". Where these tools fit in the pipeline is the subject of the DevSecOps series.

Terraform has its own policy tool too: Terraform Policy became GA with 1.17 and brought the -policies flag to the plan, apply and query commands. Rules are written in HCL in .policy.hcl files (a resource_policy block with enforce inside). According to its documentation, the rules are mainly enforced in HCP Terraform, and there is a separate tfpolicy CLI for local testing. When I tried it on a local plan:

$ terraform plan -policies=policies
...
Error: Failed to connect to policy engine

Failed to connect to policy engine: failed to connect to plugin: exec:
"tfpolicy-plugin": executable file not found in $PATH.

The policy engine runs as a separate plugin (tfpolicy-plugin) that has to be on the PATH. The local testing tool tfpolicy is distributed under a separate evaluation licence; that is why I did not install and run it, and this part is based on the documentation.

Running Everything with One Command

To run these layers the same way in a pull request and on a laptop, I wrote a script. It does not need cloud credentials; it runs with init -backend=false without connecting to the backend:

#!/usr/bin/env bash
# Checks every Terraform root and module without cloud credentials
set -euo pipefail
cd "$(dirname "$0")/../infra"

echo "== fmt"
terraform fmt -check -recursive

echo "== validate"
for dir in bootstrap-local bootstrap-azure local/envs/dev local/envs/staging azure/envs/dev azure/envs/staging azure/envs/prod; do
    terraform -chdir="$dir" init -backend=false -input=false -no-color >/dev/null
    printf '%-20s ' "$dir"
    terraform -chdir="$dir" validate -no-color
done

echo "== tflint"
tflint --init >/dev/null
tflint --recursive --config "$PWD/.tflint.hcl"

echo "== test (mock providers)"
for dir in modules/bookstore-webapp azure/bookstore; do
    terraform -chdir="$dir" init -input=false -no-color >/dev/null
    terraform -chdir="$dir" test -no-color
done

$ ./scripts/check-infra.sh
== fmt
== validate
bootstrap-local      Success! The configuration is valid.

bootstrap-azure      Success! The configuration is valid.

local/envs/dev       Success! The configuration is valid.

local/envs/staging   Success! The configuration is valid.

azure/envs/dev       Success! The configuration is valid.

azure/envs/staging   Success! The configuration is valid.

azure/envs/prod      Success! The configuration is valid.

== tflint
== test (mock providers)
tests/webapp.tftest.hcl... in progress
  run "https_and_tls"... pass
  run "open_without_ip_ranges"... pass
  run "deny_by_default_with_ip_ranges"... pass
tests/webapp.tftest.hcl... tearing down
tests/webapp.tftest.hcl... pass

Success! 3 passed, 0 failed.
tests/bookstore.tftest.hcl... in progress
  run "dev_uses_small_tiers"... pass
  run "prod_rejects_b1"... pass
  run "prod_uses_zone_redundancy"... pass
tests/bookstore.tftest.hcl... tearing down
tests/bookstore.tftest.hcl... pass

Success! 3 passed, 0 failed.

I left the bookstore-service test, which needs real Docker, out of this list; it runs separately in a job that has Docker. In post 11 this script becomes the first job of the pull request pipeline.

Production Notes

If you test with the real provider, use a separate test subscription and generate test resource names that cannot clash (for example a suffix derived from the run ID). If a test fails, the teardown can also stop halfway; without a job that regularly cleans the test subscription and a budget alert, real apply tests lead to expensive surprises.

Keep tests next to the modules and run them on every pull request that changes a module. Mocked tests take seconds, so there is no cost reason not to. Real apply tests can be enough at night, or only when a new module version is tagged.

Pin the versions of the scanners and of tflint too. A new version brings new rules, and a pull request that passed yesterday can break today without any code change. Upgrading a rule set should be a separate, deliberate change.

Common Pitfalls

Relying on validate alone. The TLS change at the start of this post and the prod rule from post 7 both passed validate and were only caught in a plan or a test. validate says "this code can run", not "this code is right".

Using the real prod subscription or the prod state in tests. A test that runs with the real provider creates and deletes resources; as we saw above, the teardown does not stop at prevent_destroy. A test should never see prod's credentials.

Mistaking a mocked test for an integration test. The mock provider accepts everything; errors such as a SKU not being available in the region, name clashes or a missing role assignment only show up in a real apply.

A Small Experiment

★☆☆ Add another assertion to the bookstore-webapp test that fails when https_only = false, and see the test really break.

★★☆ Add a run to the infra/azure/bookstore test that checks that storage is LRS and the database is Basic in the staging environment. Which mocks could you reuse?

★★★ Write the Trivy findings you accept into a .trivyignore file with their reasons and confirm that the scan shows only the findings to fix. Is it better to silence a finding in the code with a comment next to the resource, or to keep it in a separate file?

In the next post we move these checks into GitHub Actions: fmt, validate, tests and plan on the pull request; writing the plan summary to the PR as a comment, secretless Azure login with OIDC, approved apply and a scheduled drift check.