Terraform Change Control

Why infrastructure changes route through a single writer, apply before merge, and keep the configuration a static record

Related Concepts: Continuous Delivery | Acceptance Testing | Terraform State Boundaries | Implements: Terraform Orchestration, Terraform Testing

Concurrent Work on Shared State

Terraform's native state locking protects one state file for the duration of one operation, which puts it below the level of a pull request. Two engineers with open PRs against the same root module can each produce a clean plan. Whichever change applies second runs against infrastructure the first PR already modified, so the second plan is stale at the moment it applies and the apply can clobber or revert the first change. This has been a problem since Terraform's inception, and it is why a coordination layer above state locking exists at all.

The Single Writer

Every rule in our Terraform workflow follows from one property: exactly one writer changes a given piece of infrastructure at a time. Orchestration tooling supplies that property with a lock held from the first plan until the pull request closes, so reviewed changes apply in the order they were reviewed, each against current infrastructure.

Other mechanisms supply the same property when tooling is absent or unusable:

• Staffing supplies it during bootstrapping. One assigned person owns every apply, so a second writer never exists and manual commands stay safe.
• An announcement supplies it during a manual intervention. Whoever handles state corruption or a badly behaved provider resource halts other Terraform work, holds the property by hand, and releases it when they give the all clear.

Reading the workflow this way sorts the acceptable deviations from the harmful ones. A deviation that preserves a single writer changes the mechanism and keeps the guarantee. A deviation that introduces a second writer breaks the property everything else rests on, whether that second writer is a laptop running apply, someone editing the cloud console, or an automated service patching attributes Terraform manages. The damage from a second writer surfaces later as drift, a failed apply, or a fix that gets reverted.

Applying Before Merge

A plan and CI validation can both succeed while the apply still fails. Infrastructure is stateful and can change between plan time and apply time, and a large class of failures appears only when the provider attempts the real change: IAM permission errors, quota and service limits, eventual consistency, provider-side validation, and resources modified out of band. We call this a flaky apply. The name echoes flaky tests (see Acceptance Testing), and the mechanism is the same, since statefulness introduces non-determinism that degrades the reliability of the system being changed.

That failure class is the argument for applying inside the pull request after approval. The failure stays on the branch, where the contributor iterates until the apply succeeds, and the default branch holds a record of infrastructure that has actually been applied. A post-merge apply inverts the relationship: when it fails, main describes infrastructure that does not exist, and recovery takes a revert or a follow-up PR while the environment sits half applied.

The workflow charges for this. The lock lives for the life of the PR, so it depends on small, short-lived PRs and a prompt merge once the apply succeeds. Root module size appears here as lock contention, since smaller per-service root modules mean most PRs touch different modules and hold nothing another contributor needs. Terraform State Boundaries covers the same size question as it appears in plan time and coupling.

Terraform as a Static Record

The workflow assumes the configuration describes the infrastructure accurately. Anything that changes out of band introduces drift, which makes plans noisy and raises the chance an apply fails or quietly reverts a legitimate change. The principle we apply is that anything likely to change out of band belongs in its own orchestration layer, such as a deploy pipeline or an autoscaler, while Terraform keeps ownership of what holds still. ECS container definitions change on every deploy, which is why the deployment pipeline owns them.

Every attribute has exactly one owner. Two systems managing the same attribute fight each other indefinitely. A security remediation service patches a resource through the cloud API, the next plan reads that patch as drift, the apply reverts it, and the service patches it again. Naming the owner ends the fight, and ignore_changes is how the configuration records that ownership sits elsewhere. Reaching for ignore_changes on any noisy plan silences a symptom while the drift remains, so a noisy plan is worth reading as a question about who owns the attribute.

Drift Deserves Review

Detecting drift is safe to automate. Reconciling it automatically is dangerous. Drift-detection platforms commonly offer scheduled applies that revert drift with auto-approve and with nobody reading the plan. When a drifted attribute requires replacement, that reconciliation destroys and recreates the resource, and a production database recreated to fix a manual settings change is an outage.

Our response to detected drift is opening an issue. An engineer determines how the drift occurred, the issue creates an audit trail showing the remediation was intentional, and when a person caused the drift, the conversation redirects their future changes into the workflow. Preventing drift is mostly a matter of organizational process and diligence, and detection tells us when prevention failed.

The Cost of Testing Infrastructure

Infrastructure tests carry a property that shapes every decision about them: some of them provision real resources, which costs real money and real time. A mocked unit test runs without credentials and returns in seconds. An integration test creates and destroys live infrastructure, and depending on the module the bill is significant. We prefer mocked unit tests for that reason and keep integration tests thin, reserving them for behavior a mock cannot reproduce, such as IAM policy evaluation and provider-side validation. The preference follows the state of the tooling and may change as mocking matures.

Cost climbs with scope. End-to-end tests exercise several modules together, and environment E2E tests walk an entire account's dependency tree. Both are expensive to run and to maintain, so they earn a place when the interaction between modules is itself the thing being validated. Cheap tests also serve the pre-merge workflow directly, because a module regression caught by a unit test never reaches a plan, and never reaches an apply holding a lock.