Bringing Existing Infrastructure into Terraform: import Blocks, terraform query and list, moved and removed Blocks

Version note: The examples in this post were run with Terraform 1.17.2, the kreuzwerker/docker provider 4.7.0, hashicorp/http 3.6.3, PostgreSQL 18 for the pg backend and Docker Engine 29. For the AzureRM 5.14.0 code, init and validate were run; terraform query stopped at authentication without an Azure account, and no plan/apply was made against Azure.

Guess First

In post 7 we wrote the envs/staging folder for the staging environment of Bookstore (the online bookstore used as the shared example across this site's series) but did not apply it. Meanwhile Emre built staging by hand so the test team would not have to wait: a network, two volumes and the API container, all with the names Terraform would produce. Now you run terraform apply in envs/staging. Terraform does not know these resources exist. What happens: does everything fail, or does part of it quietly report "created"?

A Hand-Built Staging

On Azure this is done by clicking in the portal; in the local example the equivalent is the Docker CLI. The commands Emre ran:

$ docker network create tfseries-staging
$ docker volume create tfseries-staging-covers
$ docker volume create tfseries-staging-exports
$ docker run -d --name tfseries-staging-api --network tfseries-staging \
    -p 18481:8080 --restart unless-stopped \
    -v tfseries-staging-covers:/data/covers -v tfseries-staging-exports:/data/exports \
    -e Bookstore__Environment=staging -e Bookstore__Payments__ApiKey="$PAYMENTS_API_KEY" \
    bookstore-api:1.1.0
$ curl -s localhost:18481/health
{"status":"Healthy","environment":"staging","version":"1.1.0"}

The container is attached only to the tfseries-staging network; Terraform's code also adds the shared tfseries-shared network. We will see that difference in the plan shortly. First the answer to the guess; I ran the apply in a trial with a separate state:

$ terraform apply
...
module.bookstore.module.api.docker_volume.data["exports"]: Creating...
module.bookstore.docker_network.bookstore: Creating...
module.bookstore.module.api.docker_volume.data["covers"]: Creating...
module.bookstore.module.api.docker_volume.data["covers"]: Creation complete after 0s [id=tfseries-staging-covers]
module.bookstore.module.api.docker_volume.data["exports"]: Creation complete after 0s [id=tfseries-staging-exports]

Error: Unable to create network: Error response from daemon: network with name tfseries-staging already exists

The result is mixed. For the network Docker says "that name already exists" and the apply stops; the container never gets its turn. The volumes, however, were "created". In reality nothing was created: docker volume create called with an existing name returns the existing volume without an error, and the provider takes it for a new resource. Terraform now believes it created Emre's volumes itself and writes them to state together with their labels. The next plan -refresh-only sees that the labels are not actually there:

$ terraform plan -refresh-only
...
Note: Objects have changed outside of Terraform
...
      - labels {

In Docker, changing a volume label means a replace; the next normal plan would have wanted to delete and recreate these volumes. The way to bring existing infrastructure into code is not to "apply on top of it" but to tell Terraform explicitly which resource belongs to which address: the import block. I deleted the trial's state; Emre's resources are in place.

import Blocks

In post 4 we saw the terraform import command for a single resource. The command changes state immediately and skips review. Since Terraform 1.5 there is an import block that does the same job in code: it shows up in the plan, is reviewed in a pull request and takes effect with apply. The block has two parts: to is the resource's address in code, and id is the identifier the provider uses to find the resource in the real world. The Docker provider wants Docker's ID for networks and containers, and the name for volumes:

$ docker network inspect -f '{{.Id}}' tfseries-staging
26b2d9567429a440a7330adba171c447bc0d3cfc1a1891fe37fab513f63e8455
$ docker inspect -f '{{.Id}}' tfseries-staging-api
b356423113bbc5023fe6bb4ab5668be64e5b4dd37486cfe2685ed7fb7ff7d5fd

# Staging was built by hand with the Docker CLI; these blocks adopt it
import {
  to = module.bookstore.docker_network.bookstore
  id = "26b2d9567429a440a7330adba171c447bc0d3cfc1a1891fe37fab513f63e8455"
}

import {
  for_each = toset(["covers", "exports"])
  to       = module.bookstore.module.api.docker_volume.data[each.key]
  id       = "tfseries-staging-${each.key}"
}

import {
  to = module.bookstore.module.api.docker_container.this
  id = "b356423113bbc5023fe6bb4ab5668be64e5b4dd37486cfe2685ed7fb7ff7d5fd"
}

Two details. to can point at an address inside a module; we do not need to pull the resources out of the bookstore module. One block was enough for the volumes: since Terraform 1.7 an import block accepts for_each, and you can build the key in to with each.key.

Reading the First Plan

The most important step of an import is the first plan. Terraform takes the resource into state, then compares it with the code and tells you what it will do to close the gaps:

$ terraform plan
...
  # module.bookstore.docker_network.bookstore must be replaced
  # (imported from "26b2d9567429a440a7330adba171c447bc0d3cfc1a1891fe37fab513f63e8455")
  # Warning: this will destroy the imported resource
...
      + labels { # forces replacement
...
  # module.bookstore.module.api.docker_volume.data["covers"] must be replaced
  # (imported from "tfseries-staging-covers")
  # Warning: this will destroy the imported resource
...
      + labels { # forces replacement
...
Plan: 3 to import, 4 to add, 0 to change, 3 to destroy.
...
Error: Instance cannot be destroyed

The line "Warning: this will destroy the imported resource" is a warning to take seriously: the imported resource will be deleted in the first minute of the apply. The cause is the labels: our module puts app, environment and managed_by labels on networks and volumes, the hand-built ones have none, and Docker cannot change labels in place. For the network that is acceptable, there is no data in it. For the volumes it is not: the cover images in staging would be deleted. This time prevent_destroy protected them, but not every resource has that protection.

For every "will change" or "will replace" line after an import there are two options: make the code match reality, or accept the change knowingly. For the volumes I changed the code. Since Docker cannot update volume labels in place anyway, recreating a data volume because of a label change is something no environment wants. In the bookstore-service module:

resource "docker_volume" "data" {
  # ...
  lifecycle {
    prevent_destroy = true

    # Docker cannot change a volume's labels in place; never replace a data volume for a label
    ignore_changes = [labels]
  }
}

In post 5 we said ignore_changes should not be used to hide drift; this use is different: we deliberately leave out a field that cannot change. The second plan:

$ terraform plan
...
  # module.bookstore.module.api.docker_container.this must be replaced
  # (imported from "b356423113bbc5023fe6bb4ab5668be64e5b4dd37486cfe2685ed7fb7ff7d5fd")
  # Warning: this will destroy the imported resource
...
      + env                                         = (sensitive value) # forces replacement
...
      ~ network_mode                                = "tfseries-staging" -> "bridge" # forces replacement
...
      + networks_advanced { # forces replacement
...
      + ports { # forces replacement
...
      + volumes { # forces replacement
...
  # module.bookstore.module.api.docker_volume.data["covers"] will be imported
...
  # module.bookstore.module.api.docker_volume.data["exports"] will be imported
...
Plan: 4 to import, 3 to add, 0 to change, 2 to destroy.

The volumes are now imported unchanged. For the container the plan says replace, and the list contains fields I did not expect: env, ports, volumes. The Docker provider does not read these fields into state when it imports a container, so compared with the code they all look "new". The real differences are networks_advanced (the shared network is missing) and the labels. The container is stateless and the data lives in the volumes; I accepted the replace. The network will also be recreated because of the labels, and I accepted that too. The apply:

$ terraform apply
...
module.bookstore.module.api.docker_volume.data["covers"]: Import complete [id=tfseries-staging-covers]
module.bookstore.module.api.docker_volume.data["exports"]: Import complete [id=tfseries-staging-exports]
module.bookstore.docker_network.bookstore: Import complete [id=26b2d9567429...]
module.bookstore.module.api.docker_container.this: Import complete [id=b356423113bb...]
module.bookstore.docker_network.bookstore: Destroying... [id=26b2d9567429...]
...
module.bookstore.module.api.docker_container.this: Destroying... [id=b356423113bb...]
...
module.bookstore.module.api.docker_container.this: Creation complete after 0s [id=4fa8d12de663...]
Apply complete! Resources: 4 imported, 3 added, 0 changed, 2 destroyed.
$ curl -s localhost:18481/health
{"status":"Healthy","environment":"staging","version":"1.1.0"}
$ terraform plan
...
Plan: 0 to add, 0 to change, 0 to destroy.

First the four resources were taken into state, then the replaces from the plan were carried out. The volumes are in place, and so is their data. After the apply the import blocks no longer do anything; the documentation leaves it up to you to delete them or keep them as a record of where the resource came from. Our IDs are no longer valid because the network and the container were recreated; I deleted imports.tf.

Generating a Skeleton with -generate-config-out

In our case the code was ready. When importing a resource that has no code yet, Terraform can write its blocks for you. In an empty folder with two import blocks in the root module:

import {
  to = docker_network.staging
  id = "26b2d9567429a440a7330adba171c447bc0d3cfc1a1891fe37fab513f63e8455"
}

import {
  to = docker_container.staging_api
  id = "b356423113bbc5023fe6bb4ab5668be64e5b4dd37486cfe2685ed7fb7ff7d5fd"
}

$ terraform plan -generate-config-out=generated.tf
...
  # docker_container.staging_api must be replaced
...
      + env                                         = (known after apply) # forces replacement
...
Plan: 2 to import, 1 to add, 0 to change, 1 to destroy.

Warning: Config generation is experimental

Generating configuration during import is currently experimental, and the
generated configuration format may change in future versions.

A section from the container part of the generated file:

# __generated__ by Terraform
# Please review these resources and move them into your main configuration files.

resource "docker_container" "staging_api" {
  ...
  entrypoint                                  = ["dotnet", "Bookstore.Api.dll"]
  ...
  hostname                                    = "b356423113bb"
  image                                       = "bookstore-api:1.1.0"
  ...
  name                                        = "tfseries-staging-api"
  network_mode                                = "tfseries-staging"
  ...
  restart                                     = "unless-stopped"
  ...
  user                                        = "1654"
  ...
  working_dir                                 = "/app"
}

An 84-line file came out and the plan still wants to replace the container, because the generated code has no env, port or volume definitions, while it does have values Docker assigned itself, such as hostname. This output is a skeleton: it shows which arguments exist, but it is not code to commit as it is. Delete the unnecessary fields, add the missing ones, tie the values to variables and locals, and if possible turn it into a call to an existing module. Config generation has existed since Terraform 1.5 and still comes with an "experimental" warning; the format of the generated file may change between versions.

terraform query and list Blocks

To import, you need to know the resource's ID. If you do not know what is in a subscription that has accumulated portal-built resources over the years, terraform query, introduced in Terraform 1.14, does the discovery step. You write list blocks in files with the .tfquery.hcl extension; each block queries one resource type and lists the results the provider returns. With -generate-config-out it also produces import and resource blocks for every result.

The feature depends on provider support. I looked at the schemas of the providers I had installed: AzureRM 5.14 supports listing for 127 resource types; resource groups, storage accounts, storage containers, SQL Servers and databases, service plans, user-assigned identities and Key Vaults are among them, azurerm_linux_web_app is not. kreuzwerker/docker 4.7 has none, so I could not try it in the local example. The query I wrote for staging on the Azure side:

# Lists what was built by hand in the portal for staging
list "azurerm_resource_group" "all" {
  provider = azurerm
  config {}
}

list "azurerm_storage_account" "staging" {
  provider = azurerm
  config {
    resource_group_name = "rg-bookstore-staging-weu"
  }
}

list "azurerm_mssql_server" "staging" {
  provider = azurerm
  config {
    resource_group_name = "rg-bookstore-staging-weu"
  }
}

The resource group list can be filtered by tag, but I assume hand-built resources have no tags and list them all. I ran it in a local copy with the backend disabled; without the Azure CLI and an account, the provider stopped at authentication:

$ terraform query

Error: unable to build authorizer for Resource Manager API: could not configure AzureCli Authorizer: could not parse Azure CLI version: launching Azure CLI: exec: "az": executable file not found in $PATH

  with provider["registry.terraform.io/hashicorp/azurerm"],
  on providers.tf line 1, in provider "azurerm":
   1: provider "azurerm" {

With an account the output would be one line per resource with the resource's identity. Thanks to resource identity, introduced in Terraform 1.12, import blocks can also be written with meaningful fields instead of long Azure IDs. The blocks the query would generate look like this; written by hand, they passed validate:

import {
  to = module.bookstore.azurerm_resource_group.bookstore
  identity = {
    subscription_id = "00000000-0000-0000-0000-000000000000"
    name            = "rg-bookstore-staging-weu"
  }
}

import {
  to = module.bookstore.azurerm_storage_account.bookstore
  identity = {
    subscription_id     = "00000000-0000-0000-0000-000000000000"
    resource_group_name = "rg-bookstore-staging-weu"
    name                = "stbookstorestagingweu"
  }
}

import {
  to = module.bookstore.azurerm_mssql_server.bookstore
  id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-bookstore-staging-weu/providers/Microsoft.Sql/servers/sql-bookstore-staging-weu"
}

The first two blocks use identity, the last one the classic id. The provider schema tells you which fields make up the identity: subscription and name for a resource group, plus the resource group name for a storage account. With 1.17, terraform query can evaluate the discovered resources against policies when running remotely in HCP Terraform; the local CLI has no such step.

When Should You Delete moved Blocks?

We left two moved.tf files behind in posts 6 and 7. A moved block is there so that everyone whose state still has the old address can apply. HashiCorp's refactoring documentation treats deleting the block as a breaking change: a state still at the old address, planned without the block, tries to delete and recreate instead of moving. For published modules that others use, the documentation recommends never deleting old moved blocks. For modules within a team, deleting is safe, but only if you are sure every state has been applied with the new version.

In the local example the only state carrying old addresses is dev's, and dev has applied both changes; staging started directly with the new addresses. I deleted both files:

$ rm infra/local/bookstore/moved.tf infra/local/envs/dev/moved.tf
$ cd infra/local/envs/dev
$ terraform plan
...
Plan: 0 to add, 0 to change, 0 to destroy.

I am keeping the moved.tf files on the Azure side: someone who applied this code in their own subscription as it was in post 6 can only update it safely because of them. Base the decision to delete on "when was the last apply", not on wanting the code to look tidy.

Let us also come back to the pitfall from post 3: turning volumes created with count into for_each. In a separate trial folder I set up the first version from post 3:

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 4.7"
    }
  }
}

provider "docker" {}

variable "volume_names" {
  type    = list(string)
  default = ["covers", "exports", "invoices"]
}

resource "docker_volume" "data" {
  count = length(var.volume_names)
  name  = "tfseries-lab-${var.volume_names[count.index]}"
}

After changing the variable to set(string) and the resource to for_each, the plan:

$ terraform plan
...
  # docker_volume.data[0] will be destroyed
  # (because resource does not use count)
  # docker_volume.data[1] will be destroyed
  # (because resource does not use count)
  # docker_volume.data[2] will be destroyed
  # (because resource does not use count)
  # docker_volume.data["covers"] will be created
  # docker_volume.data["exports"] will be created
  # docker_volume.data["invoices"] will be created
Plan: 3 to add, 0 to change, 3 to destroy.

Because the addresses change from [0] to ["covers"], Terraform wants to delete and recreate the three volumes; their data would be gone. With moved blocks that map each position to a key:

# count used positions; for_each uses keys
moved {
  from = docker_volume.data[0]
  to   = docker_volume.data["covers"]
}

moved {
  from = docker_volume.data[1]
  to   = docker_volume.data["exports"]
}

moved {
  from = docker_volume.data[2]
  to   = docker_volume.data["invoices"]
}

$ terraform plan
...
  # docker_volume.data[0] has moved to docker_volume.data["covers"]
  # docker_volume.data[1] has moved to docker_volume.data["exports"]
  # docker_volume.data[2] has moved to docker_volume.data["invoices"]
Plan: 0 to add, 0 to change, 0 to destroy.

Since a moved block does not accept for_each, you write one block per element. You could write a small script that reads the list and generates these blocks, but checking the mapping between positions and keys by eye once is a must: count.index follows the order of the list, not the order of the set.

The removed Block: Letting Go Without Deleting

Sometimes you want to take a resource out of Terraform's management without deleting it: it is moving to another team or, as in post 7, to another layer's state. Deleting the resource block from the code does not do that; as we saw in post 5, when the block goes, Terraform deletes the resource. The right tool is the removed block. In the same trial folder I wanted to hand the volumes over to a "data" folder, and first tried to take out only invoices:

removed {
  from = docker_volume.data["invoices"]

  lifecycle {
    destroy = false
  }
}

$ terraform validate

Error: Resource instance keys not allowed

  on removed.tf line 2, in removed:
   2:   from = docker_volume.data["invoices"]

Resource address must be a resource (e.g. "test_instance.foo"), not a
resource instance (e.g. "test_instance.foo[1]").

removed targets the whole resource, not a single instance. If you need to split off one key, first move it to its own resource address with moved, then let go of that address with removed in a second step. I handed over all three volumes together: I deleted the resource block and put this in its place:

# The volumes now belong to the data layer; keep them, just stop managing them here
removed {
  from = docker_volume.data

  lifecycle {
    destroy = false
  }
}

$ terraform plan
...
 # docker_volume.data["covers"] will no longer be managed by Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)
 . resource "docker_volume" "data" {
        id          = "tfseries-lab-covers"
        name        = "tfseries-lab-covers"
        # (3 unchanged attributes hidden)
    }
...
Plan: 0 to add, 0 to change, 0 to destroy.

Warning: Some objects will no longer be managed by Terraform

If you apply this plan, Terraform will discard its tracking information for
the following objects, but it will not delete them:
 - docker_volume.data["exports"]
 - docker_volume.data["covers"]
 - docker_volume.data["invoices"]

After applying this plan, Terraform will no longer manage these objects. You
will need to import them into Terraform to manage them again.

The dot symbol and the warning in the plan say clearly that the resource will not be deleted but will leave the state. After the apply the volumes are still in Docker and the app folder's state is empty. The data folder that takes them over picks them up with an import that uses for_each:

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 4.7"
    }
  }
}

provider "docker" {}

locals {
  data_sets = toset(["covers", "exports", "invoices"])
}

import {
  for_each = local.data_sets
  to       = docker_volume.data[each.key]
  id       = "tfseries-lab-${each.key}"
}

resource "docker_volume" "data" {
  for_each = local.data_sets
  name     = "tfseries-lab-${each.key}"

  lifecycle {
    prevent_destroy = true
  }
}

$ terraform plan
...
  # docker_volume.data["covers"] will be imported
  # docker_volume.data["exports"] will be imported
  # docker_volume.data["invoices"] will be imported
Plan: 3 to import, 0 to add, 0 to change, 0 to destroy.
$ terraform apply
...
Apply complete! Resources: 3 imported, 0 added, 0 changed, 0 destroyed.
$ cd ../app && terraform plan
No changes. Your infrastructure matches the configuration.

These two steps are the code-based, reviewable version of "moving a resource from one state to another", which used to be done by hand with the terraform state rm and terraform import commands. The removed block can also be deleted after the apply; once the resource is no longer in state, the block has nothing left to do. But the destroy = false warning from post 5 applies here too: if you delete the block before the apply and the resource block is also gone, the plan goes back to deleting the resource.

Production Notes

Before applying an import to production, try the same flow on a copy: take the plan, write down the reason for every replace and update line, and put the ones you accept in the pull request description. On Azure, tag differences are usually in-place updates, but fields like SKU, region and name mean a replace; replacing a SQL database is data loss. Putting prevent_destroy on the data resources you are about to import stops an unexpected replace at plan time, as it did for us.

Import is not a one-off job; it is a continuation of drift control. To see resources that were never brought into code or have drifted from it, you need plan -refresh-only (post 4) and a scheduled drift job (post 11); on Azure, regularly listing "resource groups Terraform does not know about" with terraform query is also a good signal.

Watch the secrets of hand-built resources. If the settings of a resource you import contain secrets (like a Web App's app settings), those values are read into state; in our container the payments API key is in state too after the replace. If access to the state is not restricted, an import means exposing the secret to more people. The fix is the subject of the next post.

Common Pitfalls

Applying the first plan after an import blindly. If the plan does not say "0 to change", every line in between is a change to real infrastructure; if it says "Warning: this will destroy the imported resource", the resource you imported will be deleted. First bring the code closer to reality, then accept the remaining differences one by one.

Applying on top of existing resources. As we saw above, some APIs fail with "already exists", others quietly return the existing object. In the second case Terraform believes it created the resource; you end up owning a state that can delete it.

Deleting moved blocks too early, or deleting a resource block without removed. Both end the same way: instead of moving or letting go, the plan goes back to deleting. In every plan where you made a deletion decision, look at the to destroy count on the last line.

A Small Experiment

★☆☆ Stop tfseries-staging-api by hand (docker stop) and run terraform plan -refresh-only in envs/staging. Which field does Terraform say has changed?

★★☆ Create a volume named tfseries-staging-reports with the Docker CLI and import it by adding reports to the data_volumes list of bookstore-service. What did you have to change to avoid a replace in the first plan?

★★★ Try writing the moved blocks for the count to for_each switch with a for expression that reads the order of the list. Does Terraform allow it? If not, which tool would you use to generate the blocks?

In the next post we move on to secrets: what sensitive hides and what it does not, ephemeral values, write-only arguments that never write the Azure SQL password to state, and Key Vault.