The Dependency Graph, Data Sources and Lifecycle: depends_on, create_before_destroy, prevent_destroy and ignore_changes

Version note: The examples in this post were run with Terraform 1.17.2, the kreuzwerker/docker provider 4.7.0, hashicorp/http 3.6.3 and Docker Engine 29. For the AzureRM 5.14.0 code, init and validate were run; no plan/apply was made against Azure.

Guess First

You want the Bookstore API container to be renewed without downtime, so you add create_before_destroy = true to its lifecycle block: Terraform should create the new container first and delete the old one afterwards. The container's name is fixed: tfseries-dev-api. What happens on the next replace? Does the new container come up, does Terraform change the name itself, or does the apply fail? The answer is in the middle of the post, with the real error message.

The Dependency Graph: Implicit Dependencies and terraform graph

Terraform does not run blocks in the order they are written in the files. It first builds a graph from the dependencies between all resources, then walks that graph, running resources that do not depend on each other in parallel. The most common source of dependencies is references: the moment you write docker_network.bookstore.name in the container's networks_advanced block, Terraform knows the container must be created after the network. This is called an implicit dependency. The graph of this post's local example, in the Mermaid format added in Terraform 1.16:

$ terraform graph -format=mermaid
flowchart LR
  data.docker_image.api["data.docker_image.api"]
  data.docker_network.shared["data.docker_network.shared"]
  data.http.health["data.http.health"]
  docker_container.api["docker_container.api"]
  docker_network.bookstore["docker_network.bookstore"]
  docker_volume.api_data["docker_volume.api_data"]
  terraform_data.api_image["terraform_data.api_image"]
  docker_container.api --> data.docker_network.shared
  docker_container.api --> docker_network.bookstore
  docker_container.api --> docker_volume.api_data
  docker_container.api --> terraform_data.api_image
  terraform_data.api_image --> data.docker_image.api

When you paste this output into a Markdown file or a pull request description, GitHub draws it as a diagram. The default format is DOT and carries the same information. Following the arrows gives you the order: the container comes after the network, the volumes and the image have been read. The network and the volumes do not depend on each other, so they are created at the same time; remember how the volumes said "Creating..." in the same second in the apply output in post 2. By default Terraform runs ten operations at once, which you can change with -parallelism. When destroying, the graph is walked in reverse.

depends_on: A Dependency the Code Cannot See

Sometimes there is a real ordering relationship between two resources but no reference in the code. The typical example in Bookstore's Azure infrastructure is Key Vault access. The payment provider's API key lives in a shared Key Vault managed by the platform team. The Web App will read that secret with a user-assigned managed identity:

data "azurerm_key_vault" "shared" {
  name                = var.shared_key_vault.name
  resource_group_name = var.shared_key_vault.resource_group_name
}

resource "azurerm_user_assigned_identity" "api" {
  name                = "id-${local.suffix}"
  resource_group_name = azurerm_resource_group.bookstore.name
  location            = azurerm_resource_group.bookstore.location
  tags                = local.tags
}

resource "azurerm_role_assignment" "api_key_vault" {
  scope                = data.azurerm_key_vault.shared.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.api.principal_id
}

resource "azurerm_linux_web_app" "api" {
  name                = "app-${local.suffix}"
  resource_group_name = azurerm_resource_group.bookstore.name
  location            = azurerm_resource_group.bookstore.location
  service_plan_id     = azurerm_service_plan.bookstore.id
  https_only          = true
  tags                = local.tags

  app_settings = merge(local.app_settings, {
    Bookstore__Payments__ApiKey = "@Microsoft.KeyVault(VaultName=${data.azurerm_key_vault.shared.name};SecretName=payments-api-key)"
  })

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.api.id]
  }

  key_vault_reference_identity_id = azurerm_user_assigned_identity.api.id

  site_config {
    ...
  }

  # The app reads secrets at startup; nothing in this block references the role assignment
  depends_on = [azurerm_role_assignment.api_key_vault]

  lifecycle {
    # The deployment pipeline sets the image tag (CI/CD series); Terraform owns everything else
    ignore_changes = [site_config[0].application_stack[0].docker_image_name]

    precondition {
      condition     = var.environment != "prod" || var.api.sku_name != "B1"
      error_message = "Production needs at least a P0v3 plan, not B1."
    }
  }
}

Because the Web App references the identity, it is created after the identity. But the Web App does not reference the role assignment anywhere; in Terraform's eyes the two are independent and can be created in parallel. In that case the Web App can come up before the role assignment, try to resolve the @Microsoft.KeyVault(...) reference at startup and fail to get the secret because it has no permission. depends_on tells Terraform about this invisible dependency. I did not run this code on Azure; validate passed.

Use depends_on only when there is a real ordering need like this. A depends_on added just in case reduces the parallelism in the graph. On data sources the cost is higher: according to the Terraform documentation, if a resource in a data source's depends_on has a pending change, the data source is read during apply instead of during the plan. Every value fed from that data source then becomes (known after apply) in the plan, and the plan can tell you less.

Data Sources: Reading What Already Exists

The data block in the Key Vault example is a data source: it reads a resource that Terraform does not manage and someone else created. It reads but never changes or deletes it. A shared network, the platform team's Key Vault, or a DNS zone managed in another state are read this way. The local example has two data sources: the shared network the platform team created with docker network create tfseries-shared, and the API image we built:

data "docker_network" "shared" {
  name = var.shared_network_name

  lifecycle {
    postcondition {
      condition     = self.driver == "bridge"
      error_message = "The shared network must use the bridge driver."
    }
  }
}

data "docker_image" "api" {
  name = "bookstore-api:${var.api.image_tag}"
}

# Rebuilding the same tag changes the image ID; this resource carries that change
resource "terraform_data" "api_image" {
  input = data.docker_image.api.id
}

At the start of the plan you can see the data sources being read:

data.docker_image.api: Reading...
data.docker_network.shared: Reading...
data.docker_network.shared: Read complete after 0s [id=f2116a3d62ee0e1469796f3d9cac6146c0b7934bdabc935c35e2f10538348b88]
data.docker_image.api: Read complete after 0s [id=sha256:7e79244a7ecd0557fa579a28aadbca8e35e271336f5d1f1ff188f66930eb4f9e]

If the resource being read does not exist, the plan stops. Giving the wrong name for the shared network:

Error: Could not find docker network: Error response from daemon: network tfseries-missing not found

  with data.docker_network.shared,
  on main.tf line 3, in data "docker_network" "shared":
   3: data "docker_network" "shared" {

That is good behaviour: code that assumes someone else's resource exists stops up front, rather than running half an apply while that resource is missing.

lifecycle: create_before_destroy and prevent_destroy

The default order of a replace is to delete first and then create. create_before_destroy = true reverses that order. Trying to renew the container with -replace using the setting from the opening question, the plan shows the changed order with a new symbol:

  # docker_container.api will be replaced, as requested
+/- resource "docker_container" "api" {
...
Plan: 1 to add, 0 to change, 1 to destroy.
docker_container.api: Creating...

Error: Unable to create container: Error response from daemon: Conflict. The container name "/tfseries-dev-api" is already in use by container "1284850060aeaccace28350326fd4c6844eca43127f96f05cacadc66452cf462". You have to remove (or rename) that container to be able to reuse that name.

+/- instead of -/+: create first, then delete. But the new container could not take the old one's name and the apply stopped. That is the answer to the opening question: Terraform does not change the name itself. Because the old container stayed up there was no downtime, but there was no renewal either. The Terraform documentation says this plainly: create_before_destroy is opt-in because many resource types require unique names, and making it possible for the old and the new object to live side by side for a while is your responsibility. On Docker the fix is to make the name and the host port different for every version; on Azure it is the same, because Web App and Storage Account names are unique across all of Azure. That is why create_before_destroy is most useful on resources whose names Azure generates or that can take a random suffix.

prevent_destroy = true, on the other hand, rejects every plan that deletes or replaces the resource. In the local example we added it to the API's data volumes:

resource "docker_volume" "api_data" {
  for_each = var.volume_names
  name     = "${local.name_prefix}-${each.key}"

  dynamic "labels" {
    for_each = local.labels
    content {
      label = labels.key
      value = labels.value
    }
  }

  lifecycle {
    prevent_destroy = true
  }
}

$ terraform plan -destroy -var-file=dev.tfvars
...
Error: Instance cannot be destroyed

  on main.tf line 35:
  35: resource "docker_volume" "api_data" {

Resource docker_volume.api_data["exports"] has lifecycle.prevent_destroy set,
but the plan calls for this resource to be destroyed. To avoid this error and
continue with the plan, either disable lifecycle.prevent_destroy or reduce
the scope of the plan using the -target option.

The same error came when I accidentally passed the environment as prod and tried to change the volume names: a name change means a replace, and prevent_destroy stops replaces too. On the Azure side we put the same protection on BookstoreDb:

resource "azurerm_mssql_database" "bookstore" {
  name      = "BookstoreDb"
  server_id = azurerm_mssql_server.bookstore.id
  sku_name  = var.environment == "prod" ? "S1" : "Basic"
  tags      = local.tags

  lifecycle {
    prevent_destroy = true
  }
}

check "api_health" {
  data "http" "health" {
    url = "https://${azurerm_linux_web_app.api.default_hostname}/health"
  }

  assert {
    condition     = data.http.health.status_code == 200
    error_message = "The Bookstore API health endpoint returned ${data.http.health.status_code}."
  }
}

There is an important limit: prevent_destroy works only while the block itself is in the code. If you delete the resource from the code together with its block, the protection goes with it and Terraform deletes the resource.

ignore_changes and replace_triggered_by

ignore_changes makes Terraform disregard differences in an attribute. During an incident, the operations team set the container's memory limit with docker update --memory 512m --memory-swap 512m tfseries-dev-api and manages that setting with its own tooling. The next plan:

  # docker_container.api will be updated in-place
      - memory                                      = 512 -> null
      - memory_swap                                 = 512 -> null
Plan: 0 to add, 1 to change, 0 to destroy.

Terraform wants to revert the setting. After adding the two attributes to ignore_changes, the plan no longer touches the container:

resource "docker_container" "api" {
  ...
  networks_advanced {
    name = data.docker_network.shared.name
  }

  ports {
    internal = 8080
    external = var.api.external_port
  }

  lifecycle {
    replace_triggered_by = [terraform_data.api_image]

    # Memory limits are tuned by the operations team with docker update
    ignore_changes = [memory, memory_swap]

    precondition {
      condition     = var.environment != "prod" || var.api.image_tag != "latest"
      error_message = "Production must run a pinned image tag, not latest."
    }
  }
}

The Azure equivalent is in the Web App above: the deployment pipeline we built in the CI/CD series changes the image tag, and we do not want Terraform to pull it back to the old tag in the code on every apply. The path site_config[0].application_stack[0].docker_image_name points to a single attribute inside nested blocks. Tags that Azure Policy adds to resources automatically are another common example; for those you can give a single key such as tags["cost-center"].

replace_triggered_by does the opposite: when another resource changes, it makes this resource be replaced. The container above runs with the bookstore-api:1.1.0 tag. If you rebuild the same tag for a hotfix, the image ID changes but the container's image attribute stays the same string; Terraform sees no reason to renew the container. The terraform_data resource carries the image ID, and replace_triggered_by ties its change to replacing the container. After rebuilding the tag, the plan:

  # docker_container.api will be replaced due to changes in replace_triggered_by
-/+ resource "docker_container" "api" {
...
  # terraform_data.api_image will be updated in-place
  ~ resource "terraform_data" "api_image" {
        id     = "4e4febef-0bcf-27b1-9112-c7fff723d808"
      ~ input  = "sha256:7e79244a7ecd0557fa579a28aadbca8e35e271336f5d1f1ff188f66930eb4f9e" -> "sha256:9339abaff659bb735d277222fdf87e0a0dd8bee1254f9373fdee0d852e049e49"
      ~ output = "sha256:7e79244a7ecd0557fa579a28aadbca8e35e271336f5d1f1ff188f66930eb4f9e" -> (known after apply)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Still, the real fix is not to reuse the same tag; the immutable tag rule we discussed in the Docker series removes this problem at the root. Keep replace_triggered_by for cases where there is no other way.

precondition, postcondition and destroy = false

The variable validations in post 3 checked a single value. precondition and postcondition check the assumptions around a resource and can look at several values and at data sources. The precondition on the container forbids the latest tag in production. When I tried it:

Error: Resource precondition failed

  on main.tf line 98, in resource "docker_container" "api":
  98:       condition     = var.environment != "prod" || var.api.image_tag != "latest"
    ├────────────────
    │ var.api.image_tag is "latest"
    │ var.environment is "prod"

Production must run a pinned image tag, not latest.

I got this output in a copy with no state; in the real dev state, environment = "prod" changed the volume names, so prevent_destroy kicked in first. A postcondition runs after the resource or data source has been read and looks at the result through self. The postcondition on the shared network data source stops us at plan time if the platform team ever recreates the network with a driver other than bridge.

Terraform 1.16 added one more setting to lifecycle: destroy = false. According to the documentation, when the resource would be destroyed, it only removes it from the state without deleting the real object. I tried it with a small terraform_data resource; terraform plan -destroy:

 # terraform_data.keep will no longer be managed by Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)
 . resource "terraform_data" "keep" {
        id     = "c4eef20a-ec27-71e6-271b-c05c822686fe"
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Warning: Some objects will no longer be managed by Terraform

But when I deleted the block from the code entirely, the plan said "will be destroyed (because terraform_data.keep is not in configuration)". The limit from prevent_destroy applies here too: the setting works only while it is in the code. The right way to take a resource out of the code while leaving it in the real world is the removed block, in post 8.

Health Checks with check Blocks

Preconditions and postconditions stop the plan. A check block does not; it warns. It verifies on every plan and apply whether the infrastructure works as you expect when viewed from outside. We call the Bookstore API's health endpoint with the hashicorp/http provider:

check "api_health" {
  data "http" "health" {
    url = "http://localhost:${var.api.external_port}/health"

    # The container needs a few seconds to start after a replace
    retry {
      attempts     = 10
      min_delay_ms = 500
      max_delay_ms = 2000
    }
  }

  assert {
    condition     = data.http.health.status_code == 200
    error_message = "The Bookstore API health endpoint returned ${data.http.health.status_code}."
  }
}

The data source inside the check is read not during the plan but at the end of the apply. The plan shows it like this:

  # data.http.health will be read during apply
  # (config will be reloaded to verify a check block)

In my first attempt there was no retry block. The check ran right after the container was replaced, before the application had started:

docker_container.api: Creation complete after 1s [id=0d6f30dfb3ca814d2c2f9f95f2f0adfa7657ee52a8c216d8b75a82371f2cee95]
data.http.health: Reading...

Warning: Error making request
...
Error making request: GET http://localhost:18480/health giving up after 1
attempt(s): Get "http://localhost:18480/health": read tcp
127.0.0.1:59541->127.0.0.1:18480: read: connection reset by peer

Apply complete! Resources: 2 added, 0 changed, 1 destroyed.

The apply finished successfully; the check only warned. In the apply after adding retry, the health call returned 200 within a second and no warning appeared. Always set max_delay_ms as well. When the API is not up at all, the check runs during the plan too, and the wait between attempts grows with every retry. With the container stopped, my plan took 152 seconds without max_delay_ms and 18 seconds with max_delay_ms = 2000; both ended with the same warning. To see the assertion itself fail, I temporarily pointed the URL to a path that does not exist (/healthz):

Warning: Check block assertion failed

  on main.tf line 107, in check "api_health":
 107:     condition     = data.http.health.status_code == 200
    ├────────────────
    │ data.http.health.status_code is 404

The Bookstore API health endpoint returned 404.

The same check on Azure goes to the Web App's default_hostname over HTTPS. In post 11 we will set up a scheduled drift job; there the warnings from checks will become a regular signal about the health of the infrastructure, even when nobody applies.

Production Notes

Put prevent_destroy on every resource that holds data: databases, storage accounts, Key Vaults. When you really need to delete one, you open a separate pull request that removes the setting; that step makes the decision to delete go through review.

Keep the ignore_changes list short, and add a comment next to every entry explaining why it is there. Every attribute on that list is a field whose drift Terraform will no longer show.

Combine checks with a scheduled job that runs somewhere that can reach Azure. Because checks do not stop the apply, their signals are only worth something when someone reads them.

Common Pitfalls

Using ignore_changes to hide drift. The way to make Terraform accept a setting changed in the portal is to update the code or revert the change, not to make Terraform look away. ignore_changes = all in particular effectively takes the resource out of Terraform's management after it is created.

Adding depends_on just in case. It reduces parallelism, defers data source reads to apply and makes the plan vaguer. If a reference already exists, depends_on is unnecessary; if not and there is a real ordering need, write the reason in a comment next to it.

Turning on create_before_destroy for resources with fixed names. As you saw above, the new resource cannot take the old one's name and the apply stops halfway. First make sure the resource's name and its other values that must be unique (port, DNS name) allow two copies to live at the same time.

A Small Experiment

★☆☆ Paste the output of terraform graph -format=mermaid into a Markdown file and preview it on GitHub. Which resources can be created in parallel?

★★☆ Add a new volume to the for_each of docker_volume.api_data. In the plan you will see the new volume being created and the container being replaced. Then remove prevent_destroy for a moment and try deleting exports from the list; how does the plan differ with and without the setting?

★★★ Rewrite the container's name and host port so they change on every replace (for example by adding the first few characters of the image ID to the name), and make create_before_destroy work this time. How will the check block learn the new port?

In the next post we turn repeated structures into modules: the input/output contract of the bookstore-webapp module, local path, git and Terraform Registry sources, version pinning, variables in module sources added in Terraform 1.15, and retiring an input with deprecated.