Separating Environments: Workspaces, Directories or tfvars? Blast Radius and Folder Layout

Version note: The examples in this post were run with Terraform 1.17.2, the kreuzwerker/docker provider 4.7.0, hashicorp/http 3.6.3, PostgreSQL 18 for the pg backend and Docker Engine 29. For the AzureRM 5.14.0 code, init and validate were run; no plan/apply was made against Azure.

Guess First

The dev environment of Bookstore, the online bookstore used as the shared example across this site's series, is up and its state lives in the PostgreSQL backend. The team wants a staging environment. You do what looks like the shortest path: write a staging.tfvars in the same folder and run terraform plan -var-file=staging.tfvars. That was exactly the ★★☆ exercise in post 3. Will the plan create new resources for staging, or will it touch dev? And what will prevent_destroy on the volumes do this time?

Another Environment's Values on the Same State

The staging.tfvars I tried:

environment = "staging"

api = {
  image_tag     = "1.1.0"
  external_port = 18481
}

The plan output:

$ terraform plan -var-file=staging.tfvars
...
  # docker_network.bookstore must be replaced
-/+ resource "docker_network" "bookstore" {
...
      ~ name         = "tfseries-dev" -> "tfseries-staging" # forces replacement
...
  # module.api.docker_volume.data["covers"] must be replaced
...
      ~ name        = "tfseries-dev-covers" -> "tfseries-staging-covers" # forces replacement
...
Plan: 3 to add, 0 to change, 3 to destroy.
...
Error: Instance cannot be destroyed

  on ../modules/bookstore-service/main.tf line 17:
  17: resource "docker_volume" "data" {

Resource module.api.docker_volume.data["covers"] has
lifecycle.prevent_destroy set, but the plan calls for this resource to be
destroyed.

Terraform does not create anything new for staging; it tries to turn dev's resources into staging. The reason is simple: there is a single state in this folder. The state has a network at the address docker_network.bookstore named tfseries-dev. The code now says tfseries-staging for the same address, and since a Docker network cannot be renamed, it needs a replace. prevent_destroy saved the volumes and the plan stopped with an error. Had it been an unprotected resource (a web app or a storage container on the Azure side), the plan would have finished cleanly, and applying it would have deleted dev and built staging in its place.

The rule that follows is the basis for the rest of this post: every environment needs its own state. Terraform code does not describe an environment, it describes what an environment should look like; the state is what decides which environment you are managing. There are three common ways to get a separate state per environment.

Option 1: CLI Workspaces

Terraform CLI workspaces let you keep several states in the same folder and the same backend. Every folder starts with a workspace called default:

$ terraform workspace new staging
Created and switched to workspace "staging"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
$ terraform workspace list
  default
* staging

$ terraform plan -var-file=staging.tfvars
...
  # docker_network.bookstore will be created
  # module.api.docker_container.this will be created
  # module.api.docker_volume.data["covers"] will be created
  # module.api.docker_volume.data["exports"] will be created
  # module.api.terraform_data.image will be created
Plan: 5 to add, 0 to change, 0 to destroy.

This time the plan leaves dev alone and creates five new resources. Because the backend is the same, both states sit in the same PostgreSQL table, in rows separated by workspace name:

$ docker exec tfseries-state psql -U terraform -d terraform_state \
    -c "select id, name, length(data) as bytes from bookstore_dev.states order by id"
 id |  name   | bytes
----+---------+-------
  1 | default | 15834
  2 | staging |   181
(2 rows)

It is the same with the Azure Storage backend: workspaces other than default are separate blobs in the same container, with env:<workspace name> appended to the key. The advantage of workspaces is speed: one folder, one init, providers and modules downloaded once.

The problem is that only the CLI remembers which workspace you are in. To make environments differ in code you use terraform.workspace, and that easily turns into this:

locals {
  environment = terraform.workspace == "default" ? "dev" : terraform.workspace
  sku_name    = terraform.workspace == "prod" ? "P0v3" : (terraform.workspace == "staging" ? "B2" : "B1")
  replication = terraform.workspace == "prod" ? "ZRS" : "LRS"
}

If someone creating the production workspace types prd instead of prod:

$ terraform workspace new prd
$ terraform console
> [local.environment, local.sku_name, local.replication]
[
  "prd",
  "B1",
  "LRS",
]

No error at all. Production gets built with dev sizes and without zone redundancy. More importantly, all workspaces share the same backend, which means the same credentials and the same access rights: anyone who can apply to dev can also read and write the prod state. HashiCorp's documentation does not recommend workspaces for environments that need separate credentials and access control either; the use it suggests is a temporary, parallel copy to test a change. For Bookstore I keep workspaces for short-lived test environments for feature branches.

Workspaces in HCP Terraform share the name but are a different thing: each has its own variables, state and permissions, which is closer to the "directory per environment" model below.

Option 2: One Codebase, tfvars and Backend Config per Environment

In the second option the code stays in one folder and each environment gets a .tfvars file and a backend config file. The backend block is left empty in the code (the partial configuration from post 4) and the environment-specific part is passed to init:

# backend/dev.pg.tfbackend
schema_name = "bookstore_dev"

# backend/staging.pg.tfbackend
schema_name = "bookstore_staging"

$ terraform init -backend-config=backend/dev.pg.tfbackend
$ terraform plan -var-file=dev.tfvars
...
Plan: 0 to add, 0 to change, 0 to destroy.

$ terraform init -reconfigure -backend-config=backend/staging.pg.tfbackend
$ terraform plan -var-file=staging.tfvars
...
Plan: 5 to add, 0 to change, 0 to destroy.

$ terraform plan -var-file=dev.tfvars
...
  # docker_network.bookstore will be created
  # module.api.docker_container.this will be created
...
Plan: 5 to add, 0 to change, 0 to destroy.

The first two steps behave as expected: no change in dev's state, and five resources to be created in staging's empty state. The third step is the weak spot of this approach. The folder remembers the backend it was last initialised with (in .terraform/terraform.tfstate), while the tfvars are passed separately on every command. With the staging backend connected I planned with dev's values, and Terraform got ready to write dev-named resources into staging's state. Had it been applied, the network and containers with dev's names would have clashed; on Azure, a second resource with the same name either fails or, worse, ends up with two states managing the same resource.

This approach works well in CI, because the pipeline runs init and plan in every job with the same environment name. For someone switching between two environments on a laptop, though, "the backend in one place, the values in another" is a source of mistakes.

Option 3: A Directory per Environment and a Shared Module

In the third option every environment is a separate root module. The shared code moves into a module; the environment folder keeps only the backend, the provider and a module call with that environment's values. The new layout of the local example: bookstore is the module that describes one environment (network, API module, check), and each folder under envs is an environment's root.

infra/
├── bootstrap-local/
├── modules/
│   └── bookstore-service/
└── local/
    ├── bookstore/
    └── envs/
        ├── dev/
        └── staging/

The files from the old infra/local (locals, network, the API module call, the check block) moved unchanged into infra/local/bookstore; only the provider block came out, because, as we saw in post 6, provider configuration belongs in the root. The root of the dev environment:

terraform {
  required_version = "~> 1.17"

  # The connection string comes from the PG_CONN_STR environment variable
  backend "pg" {
    schema_name = "bookstore_dev"
  }

  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 4.7"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.6"
    }
  }
}

provider "docker" {}

module "bookstore" {
  source = "../../bookstore"

  environment = "dev"
  api = {
    image_tag     = "1.1.0"
    external_port = 18480
    app_settings = {
      Logging__LogLevel__Default = "Information"
    }
  }
  payments_api_key = var.payments_api_key
}

The schema name in the backend and the environment's values are now in code, in the same folder. There is no separate tfvars file; whoever wants to change dev opens a pull request that changes envs/dev, and the reviewer sees from the file path which environment is affected. The staging folder looks the same, only with the schema bookstore_staging, port 18481 and no log setting.

dev's state already exists and its addresses are written relative to the root. To avoid recreating the resources we use the moved blocks from post 6, this time one level up:

# This environment's resources used to live in the root module (infra/local)
moved {
  from = docker_network.bookstore
  to   = module.bookstore.docker_network.bookstore
}

moved {
  from = module.api
  to   = module.bookstore.module.api
}

Note the second block: it moves the whole of module.api in one block, with no need to list its four resources separately. init and plan in the new folder:

$ cd infra/local/envs/dev
$ export PG_CONN_STR=$(terraform -chdir=../../../bootstrap-local output -raw pg_conn_str)
$ terraform init
...
- bookstore in ../../bookstore
- bookstore.api in ../../../modules/bookstore-service
...
$ terraform plan
...
  # docker_network.bookstore has moved to module.bookstore.docker_network.bookstore
  # module.api.docker_container.this has moved to module.bookstore.module.api.docker_container.this
  # module.api.docker_volume.data["covers"] has moved to module.bookstore.module.api.docker_volume.data["covers"]
  # module.api.docker_volume.data["exports"] has moved to module.bookstore.module.api.docker_volume.data["exports"]
  # module.api.terraform_data.image has moved to module.bookstore.module.api.terraform_data.image
Plan: 0 to add, 0 to change, 0 to destroy.

Because we connected to the same schema, init found the existing state and the plan only shows address changes. After the apply the container is still in place and the health endpoint returns 200. The staging folder starts with its own empty state and never touches dev:

$ cd ../staging
$ terraform init
$ terraform plan
...
  # module.bookstore.docker_network.bookstore will be created
  # module.bookstore.module.api.docker_container.this will be created
  # module.bookstore.module.api.docker_volume.data["covers"] will be created
  # module.bookstore.module.api.docker_volume.data["exports"] will be created
  # module.bookstore.module.api.terraform_data.image will be created
Plan: 5 to add, 0 to change, 0 to destroy.

The cost of this layout is one init per environment and a few lines repeated across environment folders. What you get is a separate state, a separate backend setting and, if needed, separate credentials per environment, and the file path tells you which environment you are working on. HashiCorp's workspace documentation recommends this model when separate access control is needed: the shared part in a module, and each environment a separate configuration made of only a backend and a few module calls.

The Azure Side and the Three Options Compared

The Azure code moved to the same layout: the bookstore module builds an environment's resource group, storage, SQL and web app resources, and envs holds the dev, staging and prod folders.

infra/azure/
├── bookstore/
└── envs/
    ├── dev/
    ├── staging/
    └── prod/

terraform {
  required_version = "~> 1.17"

  backend "azurerm" {
    resource_group_name  = "rg-bookstore-tfstate-weu"
    storage_account_name = "stbookstoretfstateweu"
    container_name       = "tfstate"
    key                  = "bookstore/prod.tfstate"
    use_azuread_auth     = true
  }

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 5.14"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.6"
    }
  }
}

module "bookstore" {
  source = "../../bookstore"

  environment = "prod"
  api = {
    image_name = "bookstore/bookstore-api"
    image_tag  = "1.1.0"
    sku_name   = "P0v3"
  }
  shared_key_vault = {
    name                = "kv-platform-prod-weu"
    resource_group_name = "rg-platform-prod-weu"
  }
  sql_admin_password = var.sql_admin_password
}

The partial backend file from post 4 (backend/dev.azurerm.tfbackend) is no longer needed; each environment's state key is written in its own folder. Production's values are visible in code too: a P0v3 plan, no IP restriction, Key Vault kv-platform-prod-weu. The "no B1 in prod" validation we put on the root's api variable in post 6 moved along with the module; if you delete sku_name in the prod folder, validate still passes; the rule is evaluated during plan. I tried this without going to Azure, with a terraform test using mock_provider, the subject of post 10:

$ terraform test
tests/prod.tftest.hcl... in progress
  run "plan"... fail

Error: Invalid value for variable

  on main.tf line 5, in module "bookstore":
   5:   api = {
   6:     image_name = "bookstore/bookstore-api"
   7:     image_tag  = "1.1.0"
   8:   }
    ├────────────────
    │ var.api.sku_name is "B1"
    │ var.environment is "prod"

Production needs at least a P0v3 plan, not B1.

The dev folder has a moved.tf that moves the six addresses of the old root under module.bookstore. In all three folders init -backend=false and validate passed; I did not plan against Azure.

Splitting State into Layers: Blast Radius

Separating environments is one axis. The second is splitting the resources inside a single environment across several states. The goal is to shrink the blast radius: an apply that changes a web app setting should not be in a position to delete the database. A common split on Azure:

• Platform/network layer: VNet, subnets, DNS zones, the shared Key Vault. Changes rarely, often owned by the platform team.
• Data layer: Azure SQL, BookstoreDb, the storage account. Long-lived, and losing it cannot be undone.
• App layer: App Service Plan, Web App, managed identity. May change with every deploy.

When every layer has its own state, a plan that goes wrong in the app layer can at worst break the app layer; it cannot touch a database that is not in its state. The cost is having to pass data between layers: the app layer has to know the SQL Server's name and the shared Key Vault's ID. There are three ways.

We already use the first one: data sources. We read the shared Docker network and the shared Key Vault on Azure by name. As long as the naming convention is fixed, this is the loosest coupling; the reader does not need to know who built the resource or with which tool.

The second is reading another state's outputs with terraform_remote_state. In a separate folder I read dev's outputs:

data "terraform_remote_state" "dev" {
  backend = "pg"

  config = {
    conn_str    = var.pg_conn_str
    schema_name = "bookstore_dev"
  }
}

output "dev_api_url" {
  value = data.terraform_remote_state.dev.outputs.api_url
}

$ terraform apply
...
Outputs:

dev_api_url = "http://localhost:18480/health"

$ grep -o '"conn_str"[^,]*' terraform.tfstate
"conn_str": "postgres://terraform:<password>@localhost:15480/terraform_state?sslmode=disable"
...

It works, but watch two things. As the documentation warns, even though the data source only exposes outputs, the reader needs access to the whole state; in our example the same connection details can also read the payments API key in dev's state. Second, a detail I noticed while trying it: the data source's config, including the connection string and its password, was written into the reader's own state. Making the variable sensitive did not prevent that; I masked the password in the output.

The third is for a layer to publish the values it wants to share somewhere else: a Key Vault secret, Azure App Configuration, or tags on the resources. The reader then has access to those values only, not to the state. My order for Bookstore: first data sources driven by the naming convention, published values when needed, and terraform_remote_state only between layers owned by the same team at the same access level.

I did not split Bookstore into layers in this post: there are few resources and the platform layer already belongs to another team. The time to split out the data layer is when the database and the web app start being changed by different people at different rates.

Promoting a Module Version Across Environments: const Variables

Because the shared modules live in this repo, every environment uses the module as it is in the same commit; a module change that was never tried in dev is prod's code the moment it is merged. If the modules live in a separate repo, environments can sit on different versions, and a version can be promoted from dev to staging and then to prod. The Terraform 1.15 feature from post 6 is useful here: using a const variable in the module source.

variable "modules_ref" {
  type        = string
  description = "Tag of the shared modules repository used by this environment."
  const       = true
}

module "api" {
  source = "git::https://github.com/example/bookstore-terraform-modules.git//bookstore-service?ref=${var.modules_ref}"
  # ...
}

Each environment folder has a one-line terraform.tfvars, and a promotion becomes a pull request that changes that line. In my trial I pointed the same address at a local copy of the repo (file://):

$ cat terraform.tfvars
modules_ref = "v1.0.0"
$ terraform init
Downloading git::file:///home/user/repos/bookstore-terraform-modules.git?ref=v1.0.0 for api...
- api in .terraform/modules/api/bookstore-service
Terraform has been successfully initialized!

$ echo 'modules_ref = "v1.1.0"' > terraform.tfvars
$ terraform plan

Error: Module source has changed

  on main.tf line 8, in module "api":
   8:   source = "git::file:///home/user/repos/bookstore-terraform-modules.git//bookstore-service?ref=${var.modules_ref}"

The source address was changed since this module was installed. Run
"terraform init" to install all modules required by this configuration.

$ terraform init
Downloading git::file:///home/user/repos/bookstore-terraform-modules.git?ref=v1.1.0 for api...

init read the automatically loaded terraform.tfvars and downloaded v1.0.0. After changing the ref, plan refuses to run until the module is installed again, so you never get a plan that mixes the old module with the new value. That is one more reason to run init in every CI job.

Production Notes

Environment separation does not end with state; it has to cover identity too. Putting production in a separate subscription and giving the identity that can apply to prod only to the prod pipeline limits the damage of an "apply in the wrong folder" mistake. The states should be separated as well: dev and staging can share a storage account under different keys, but a separate storage account and access list for prod's state is safer, because, as we saw in post 4, state keeps secrets in plain text.

Keep the differences between environment folders small. Every difference between dev and prod (SKU, replication, IP restrictions) should be visible in the module call; if an environment needs an extra resource, add it as a separate resource in that environment's folder instead of burying an environment == "prod" condition in the module. The more alike the environments are, the better a problem seen in staging represents prod.

As the number of folders grows, so do init, provider downloads and plan time. Downloading providers once with TF_PLUGIN_CACHE_DIR, and in CI planning only the folders that changed, keeps this cost down; we will do that when we build the pipeline in post 11.

Common Pitfalls

All environments in one state. If dev, staging and prod live in the same root module with the environment name in front of the resource names, every plan reads all three environments and every apply can touch all three. A lock on one environment holds up the others too. The shortest way to separate them is separate states, as in this post.

Using the workspace name as a condition in code. As the prd example above shows, an unexpected name does not fail; it silently falls back to the default value. If you use workspaces, still pass the environment as a variable and check with a validation that it matches the workspace name.

Copy-pasting environment folders and leaving them without a module. If the same resource definitions are maintained by hand in three folders, a fix made in dev may never reach prod. An environment folder should contain a module call and values, not resources.

A Small Experiment

★☆☆ Apply envs/staging and look at the states table in the bookstore_staging schema with docker exec tfseries-state psql. How does it differ from dev's schema?

★★☆ Add a check to the workspace example that says the workspace name can only be default, staging or prod (for example a precondition on a terraform_data resource). What does plan say in the prd workspace?

★★★ Split out Bookstore's data layer: rearrange the local example so the volumes move to a separate folder and state, and the API folder finds the volume names with a data source or the naming convention. At which step did you need to take a resource out of one state and into another?

In the next post we move on to the tools that answer exactly that last question: bringing a hand-built staging into code with import blocks, finding resources with terraform query, when to delete moved blocks, and taking a resource out of Terraform's management without deleting it using removed.