Secrets in Terraform: sensitive, Ephemeral Values, Write-only Arguments and Key Vault

Version note: The examples in this post were run with Terraform 1.17.2, the kreuzwerker/docker provider 4.7.0, the cyrilgdn/postgresql provider 1.27.0, hashicorp/random 3.9.1, PostgreSQL 18 and Docker Engine 29. For the AzureRM 5.14.0 code, init and validate were run and a plan was made with a terraform test using mock_provider; no plan/apply was made against Azure.

Guess First

In the bootstrap-local folder that sets up the state backend of Bookstore (the online bookstore used as the shared example across this site's series), we expose the PostgreSQL connection string as an output with sensitive = true. terraform output shows it as <sensitive>. So in how many places does the password sit in this folder's state file? What does terraform output -json show? And in the dev environment that carries the payments API key, can someone who gets hold of a plan saved with plan -out=tfplan read the key?

sensitive Only Hides the Screen

The answer to the first two questions:

$ cd infra/bootstrap-local
$ terraform output
pg_conn_str = <sensitive>
$ terraform output -json
{
  "pg_conn_str": {
    "sensitive": true,
    "type": "string",
    "value": "postgres://terraform:<password>@localhost:15480/terraform_state?sslmode=disable"
  }
}

$ grep -n "$PASSWORD" terraform.tfstate
8:      "value": "postgres://terraform:<password>@localhost:15480/terraform_state?sslmode=disable",
55:              "POSTGRES_PASSWORD=<password>",
244:            "result": "<password>",
$ ls -l terraform.tfstate
-rw-r--r--  1 user  wheel  8424 Nov 14 20:35 terraform.tfstate

The password sits in state in plain text in three places: the value random_password generated, the container's POSTGRES_PASSWORD environment variable, and the output. The -json output also gives the value as it is; the sensitive flag is just a field in the JSON. File permissions do not protect it either: Terraform does not write local state with restricted permissions, and on this machine anyone can read the file.

The third question, the payments API key in the dev environment:

$ cd infra/local/envs/dev
$ terraform state pull | grep -c "$TF_VAR_payments_api_key"
1
$ terraform plan -out=tfplan
$ terraform show -json tfplan | grep -o "Bookstore__Payments__ApiKey=[^\"]*"
Bookstore__Payments__ApiKey=<key>
Bookstore__Payments__ApiKey=<key>

A saved plan file contains every value so that apply can run with the same values; terraform show -json gives the secret in plain text. In post 11 we will create the plan in one job and carry it to the apply job as an artifact; that artifact needs the same protection as the state.

There is one more place that gets overlooked: backend settings. If you pass the connection string with -backend-config, init writes it to .terraform/terraform.tfstate; the PG_CONN_STR environment variable we have used since post 4 is not written to any file:

$ terraform init -backend-config="conn_str=$PG_CONN_STR"
$ grep conn_str .terraform/terraform.tfstate
      "conn_str": "postgres://terraform:<password>@localhost:15480/terraform_state?sslmode=disable",

$ rm -rf .terraform && terraform init
$ grep conn_str .terraform/terraform.tfstate
      "conn_str": null,

In short, sensitive hides the value in plan and apply output and in terraform output, and does nothing else. The state, saved plans and -json output carry the full value. That is why in post 4 we talked about restricting access to the state. But restricting access is always weaker protection than not writing the secret to state at all.

Ephemeral Values

Since Terraform 1.10 some values can be "ephemeral": values Terraform uses while running but never writes to state or to a plan file. They come in three forms:

• Ephemeral variable: ephemeral = true in a variable block. The value arrives through TF_VAR_ or -var and stays in no file.
• Ephemeral resource: the ephemeral block. It is opened or read again on every run; for example ephemeral "random_password" generates a new password in memory on every plan and apply, and AzureRM's ephemeral "azurerm_key_vault_secret" reads a secret from Key Vault.
• Ephemeral output: ephemeral = true in an output block. It can only be used in child modules; it lets an ephemeral value leave a module.

The places these values can go are limited: provider configuration, another ephemeral variable or output, provisioners, and write-only arguments. You cannot pass them to a normal resource argument, because that argument is written to state. To try it, I made the payments API key in the dev environment ephemeral:

variable "payments_api_key" {
  type        = string
  description = "Key for the payment provider's sandbox."
  sensitive   = true
  ephemeral   = true
}

$ terraform validate

Error: Ephemeral value not allowed

  on main.tf line 14, in module "bookstore":
  14:   payments_api_key = var.payments_api_key

This input variable is not declared as accepting a ephemeral values, so it
cannot be set to a result derived from an ephemeral value.

An ephemeral value cannot be passed to a module input that does not explicitly say it accepts one. After making the relevant variables in the bookstore and bookstore-service modules ephemeral as well, the error moves down to where the value is actually used:

$ terraform plan
...
Error: Invalid use of ephemeral value

  with module.bookstore.module.api.docker_container.this,
  on ../../../modules/bookstore-service/main.tf line 41, in resource "docker_container" "this":
  41:   env     = local.env

Ephemeral values are not valid for "env", because it is not a write-only
attribute and must be persisted to state.

The Docker provider's env argument is not write-only; a container's environment variables have to be written to state. So in the local example there is no way on the Terraform side to keep the payments key out of state. On Azure we solve it differently: the Web App gets the key not from Terraform but from Key Vault at run time through a Key Vault reference (the @Microsoft.KeyVault(...) app setting from post 6). Terraform never sees the key's value, it only knows where it lives.

Write-only Arguments

Write-only arguments, introduced in Terraform 1.11, are where an ephemeral value can go into a resource. The provider sends the argument to the API during create and update, but Terraform does not write it to state or plan; in state its value is always null. Which resources have write-only arguments depends on the provider. I looked at the schemas of the providers I had installed: AzureRM 5.14 has seven, among them administrator_login_password_wo on azurerm_mssql_server and value_wo on azurerm_key_vault_secret. kreuzwerker/docker has none.

To see the behaviour on a real database without going to Azure SQL, I set up a separate trial folder that creates a role in our state backend's PostgreSQL. The cyrilgdn/postgresql provider's postgresql_role resource has the same pair:

terraform {
  required_version = "~> 1.17"

  required_providers {
    postgresql = {
      source  = "cyrilgdn/postgresql"
      version = "~> 1.27"
    }
  }
}

variable "admin_password" {
  type      = string
  ephemeral = true
}

variable "reporting_password" {
  type      = string
  ephemeral = true
}

variable "reporting_password_version" {
  type    = number
  default = 1
}

provider "postgresql" {
  host     = "localhost"
  port     = 15480
  database = "terraform_state"
  username = "terraform"
  password = var.admin_password
  sslmode  = "disable"
}

resource "postgresql_role" "reporting" {
  name                = "bookstore_reporting"
  login               = true
  password_wo         = var.reporting_password
  password_wo_version = var.reporting_password_version
}

Both passwords are ephemeral variables: one for the provider to connect, the other for the role's password. Here we see that ephemeral values can go into provider configuration.

$ terraform plan -out=tfplan
...
  # postgresql_role.reporting will be created
  + resource "postgresql_role" "reporting" {
...
      + login                     = true
      + name                      = "bookstore_reporting"
      + password_wo               = (write-only attribute)
      + password_wo_version       = "1"
...
$ terraform apply tfplan
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
$ grep -c "$TF_VAR_reporting_password" terraform.tfstate
0
$ terraform show -json tfplan | grep -c "$TF_VAR_reporting_password"
0
$ grep -n '"password' terraform.tfstate
29:            "password": null,
30:            "password_wo": null,
31:            "password_wo_version": "1",

The plan writes (write-only attribute) instead of the value. After the apply the password is in neither the state nor the plan file; in state the field is null. The role can connect to the database with that password.

So if Terraform does not know the value, how does it notice the password changed? It does not. That is why every write-only argument has a _wo_version argument next to it, and the password is sent only when that number changes:

$ export TF_VAR_reporting_password=<new password>
$ terraform plan
No changes. Your infrastructure matches the configuration.
$ docker exec tfseries-state psql \
    "postgresql://bookstore_reporting:$TF_VAR_reporting_password@<ip>:5432/terraform_state" \
    -tAc "select current_user"
psql: error: connection to server at "<ip>", port 5432 failed: FATAL:  password authentication failed for user "bookstore_reporting"

$ terraform apply -var reporting_password_version=2
...
      ~ password_wo_version                 = "1" -> "2"
...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
$ docker exec tfseries-state psql \
    "postgresql://bookstore_reporting:$TF_VAR_reporting_password@<ip>:5432/terraform_state" \
    -tAc "select current_user"
bookstore_reporting

I passed the new password but did not bump the version: the plan said "No changes" and the role stayed on the old password. After setting the version to 2 the update went through and the new password worked. Remembering a rotation now means bumping a version number; changing the secret and forgetting the version means believing the password changed when it did not.

The Azure SQL Password Without Ever Touching State

Since post 3 the SQL administrator password has come from a sensitive variable and sat in state in plain text. Now we want Terraform to generate the password itself and write it to SQL Server and Key Vault without anyone needing to know it. infra/azure/bookstore/secrets.tf:

# Generated in memory on every run; Terraform never writes it to state or plan
ephemeral "random_password" "sql_admin" {
  length           = 32
  special          = true
  override_special = "!#%*-_=+"
  min_lower        = 2
  min_upper        = 2
  min_numeric      = 2
  min_special      = 2
}

# Break-glass copy for people; the app connects with its managed identity, not this password
resource "azurerm_key_vault_secret" "sql_admin_password" {
  name             = "bookstore-sql-admin-password"
  key_vault_id     = data.azurerm_key_vault.shared.id
  content_type     = "password"
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = var.sql_admin_password_version
}

SQL Server now takes the password from the write-only argument:

resource "azurerm_mssql_server" "bookstore" {
  name                = "sql-${local.suffix}"
  resource_group_name = azurerm_resource_group.bookstore.name
  location            = azurerm_resource_group.bookstore.location
  version             = "12.0"
  administrator_login = var.sql_admin_login
  minimum_tls_version = "1.2"
  tags                = local.tags

  # Write-only: sent to Azure only when the version changes, never stored in state or plan
  administrator_login_password_wo         = ephemeral.random_password.sql_admin.result
  administrator_login_password_wo_version = var.sql_admin_password_version
}

The flow goes like this. On the first apply the ephemeral random_password generates a password in memory; the same value goes as write-only to both SQL Server and the Key Vault secret. On every later plan a new password is generated, but since neither resource's version changed, it is sent nowhere. To rotate, you bump sql_admin_password_version by one; in the same apply the new password goes to both places and they stay consistent. The root's sql_admin_password variable and the variables.tf in the three environment folders are gone; nobody needs to pass TF_VAR_sql_admin_password any more.

I did not apply this code to Azure. validate passed in all three environments; without going to Azure I made a plan with mock_provider, and the ephemeral resource, the write-only arguments and the Key Vault secret planned without problems. The plan's only complaint was that the Web App's health check can only be known after apply (the check block from post 5). The details of this trial method are in the next post.

This has a cost: the identity Terraform runs as needs to be able to write secrets to the platform team's Key Vault (the Key Vault Secrets Officer role). Read access is not enough. If the platform team does not want that, a separate Key Vault for Bookstore is cleaner.

Giving Secrets to the App: Key Vault and Managed Identity

The administrator password in Key Vault is a break-glass copy for people; it is not something the application should use. The right model for the Bookstore API is not to carry the secret at all: the Web App's user-assigned managed identity (the bookstore-webapp module from post 6) can read secrets from Key Vault and connect to Azure SQL with Entra ID. Terraform only builds the infrastructure here: the identity, the role assignments, the Key Vault reference. The secret's value never enters any of Terraform's files.

Azure SQL allows one more step: turning SQL authentication off completely with azuread_authentication_only = true in the azuread_administrator block. Then no administrator password is needed either; the AzureRM documentation says neither password argument is required in that case. How the application reads secrets at run time with Key Vault and managed identity, rotation and caching are the subject of the Configuration and Secret Management series.

bootstrap-local: The State Backend's Password

Back to the three copies we saw at the start. bootstrap-local manages the PostgreSQL container that runs the state backend, and the container's password goes through the env argument; as we saw above, the Docker provider has no write-only arguments. Making the password ephemeral fails, and passing it from outside does not keep it out of state. What I did and did not do for this folder:

• The state stays local and .gitignore keeps it out of the repo; as we said in post 4, this stack runs once, on the machine of whoever sets up the platform.
• The connection string reaches the other folders through PG_CONN_STR, not -backend-config; as the trial above showed, that way it is not written into the .terraform folders.
• The password only opens a container on localhost. The same design would be unacceptable on a shared server.

Its Azure counterpart (bootstrap-azure) does not have this problem, because there is no password there: shared keys are disabled on the storage account and the backend connects with Entra ID (use_azuread_auth). That is the real lesson of this post: the best secret is the one that does not exist. Wherever identity-based authentication is possible, use an identity instead of carrying a password through Terraform; where a password is unavoidable, choose resources that have write-only arguments.

Production Notes

Switching to a write-only argument does not delete the old value from state. When an existing SQL Server moves from administrator_login_password to _wo, the field becomes null in the new state version, but the old state versions kept by the backend (the blob versioning from post 4) still hold the old password. Rotate the password once after the switch so the old copies become useless.

Ephemeral resources run again on every plan and apply. An ephemeral "azurerm_key_vault_secret" that reads from Key Vault means one request to Key Vault per plan; in CI the plan job's identity must be allowed to read that secret. The permissions needed for plan can therefore get close to those of apply, and we will take that into account when we separate the identities in post 11.

Write provider and backend credentials neither into tfvars nor into code. AzureRM's credentials should come from the Azure CLI session or, in CI, from OIDC; even if a client secret is required, it should come from an environment variable such as ARM_CLIENT_SECRET.

Common Pitfalls

Printing terraform output -json to the CI log. As we saw at the start, -json gives sensitive outputs in plain text. When you pass an output to the next step in a pipeline, take only what you need with terraform output -raw <name> and register it with the platform's secret masking; do not print the whole JSON to the log.

Writing the provider credential into tfvars. terraform.tfvars is often committed, and even when it is not, it gets passed around the team. A credential that goes into a variable can also end up in state; providers can read credentials from their own environment variables.

Forgetting to bump a write-only argument's version. You changed the secret, the plan said "No changes", and you thought the rotation was done. If you automate rotation, the step that changes the secret must also bump the version number.

A Small Experiment

★☆☆ Look for the sensitive values in the output of terraform state pull in the dev environment. Which attributes of which resources carry secrets?

★★☆ In the trial folder, replace the reporting_password ephemeral variable with an ephemeral "random_password". What do you see in the second terraform plan? If nobody knows the password you generated, how would you use it?

★★★ In bootstrap-local, create a role for the backend that only has rights on the bookstore_dev schema instead of the superuser, and set its password with password_wo. Which PostgreSQL privileges did you need? How much closer did this bring you to the "separate access per environment" goal from post 7?

In the next post we move on to the layers of validating and testing Terraform code without going to Azure: fmt, validate, tflint, terraform test, the mock_provider we also used in this post, and policy checks.