Appearance

Terraform Orchestration ​

How we coordinate multiple contributors working on shared Terraform code using automation and collaboration tooling

Related Concepts: Terraform Change Control | Continuous Delivery | Terraform Project Structure | Terraform Testing

Synapse prefers orchestration tools like Atlantis or Terrateam to coordinate multiple contributors working on the same Terraform configuration. These tools belong to a category aptly named TACOS (Terraform Automation and Collaboration Software). They run plans and applies from pull requests and enforce a level of locking above Terraform's native state locking, so contributors avoid stepping on each other's work while changing infrastructure at the same time.

How the Tool Coordinates Work ​

A TACOS moves plan and apply into the pull request workflow. Terraform Change Control covers the concurrency problem this arrangement solves and the single-writer property it supplies.

  • When a PR touches a root module, the tool runs a plan and posts the output as a PR comment, so reviewers see the exact infrastructure diff alongside the code change.
  • The apply is triggered from the PR itself, typically through a comment such as atlantis apply, so the reviewed plan is exactly what gets applied.
  • The tool locks the affected root module from the first plan until the PR merges or closes. Other PRs touching the same module wait until the lock releases, which guarantees changes apply in the order they were reviewed, each against the current state of the infrastructure.

Pre-Merge Applies ​

We apply Terraform changes within the PR, after approval and before merge. The contributor iterates on the branch until the apply succeeds, and merges after that. Terraform Change Control covers the flaky apply that motivates this ordering and what a post-merge apply costs when it fails.

The workflow carries obligations:

  • Keep PRs small and short-lived, because the root module lock is held for the life of the PR.
  • Merge promptly once the apply succeeds. An applied PR that sits unmerged leaves the infrastructure ahead of the default branch and blocks other work on the same module.
  • Prefer small, per-service root modules to keep lock contention low. See Terraform Project Structure.
  • Restrict the ability to trigger an apply to the same people who can approve the PR, because an apply changes real infrastructure before anything lands on main.

The Bootstrapping Phase ​

Greenfield projects start with nothing: no state backend, no CI, and no orchestration tool. A TACOS has prerequisites of its own before it can run, such as a GitHub App installation, environment variables and credentials, and the infrastructure it runs on, and something has to create those first. We assign one person to bootstrapping, and that person runs Terraform commands from their local workstation until the bootstrapped environments are spun up and the orchestration software is in place and ready to use.

Manual Terraform interaction is allowed during this phase, since one person owns every apply. The phase ends when the orchestration tool is operational. From that point forward, manual Terraform commands stop entirely and every change routes through source control and the orchestration software.

A bootstrapping phase also applies when a project starts from existing infrastructure. Adopting a brownfield or legacy Terraform configuration usually requires intentional, in-depth refactoring before the codebase supports the structure that makes orchestration effective (see Terraform Project Structure). Importing a large volume of resources for a client project that has no IaC yet involves the same kind of sustained, hands-on state manipulation. Both are bootstrapping phases in a different shape than the traditional new-project, new-repository situation: one assigned person works with Terraform manually, and the phase ends the same way, with the orchestration tool operational and every subsequent change routed through source control.

Manual Intervention ​

Rare situations require working with Terraform outside the orchestration tool after bootstrapping is over. Examples include state corruption and provider-level resources implemented poorly enough that resolving a change requires manual terraform state mv operations or other direct intervention. When one of these situations occurs, the person handling it notifies the team to halt all Terraform work until the issue is resolved. Exactly one person touches state while the intervention is underway, and normal PR-driven work resumes once they give the all clear.

Anti-Patterns ​

Each of the following introduces a second writer to the infrastructure the tool manages. Terraform Change Control explains why that breaks the workflow and how each one surfaces later.

  • Local applies. Once the bootstrapping phase ends, the orchestration tool is the only thing that applies, outside the coordinated exceptions under Manual Intervention.
  • Console changes. Avoid fixes made directly in the cloud console (ClickOps). When an incident makes one unavoidable, codify the change back into the configuration as soon as the incident closes, while the details are still fresh.
  • Overlapping automation. Give every attribute exactly one owner, and keep a second system away from any attribute Terraform manages.
  • Auto-approved reconciliation. Leave automatic reconciliation off. Automated drift detection is fine (see Keeping Terraform Static); respond to what it finds by opening an issue and reviewing the change like any other.
  • Blanket ignore_changes. Reserve ignore_changes for attributes deliberately owned by another system. Treat any other noisy plan as a question about which system owns the attribute.

Keeping Terraform Static ​

Terraform stays a static representation of the infrastructure it manages, and out-of-band changes are what erode that. We minimize drift in a few ways:

  • Comprehensive tests. A test suite catches module-level regressions before a plan ever runs. See Terraform Testing.
  • Lifting frequently changing resources out of Terraform. ECS container definitions change on every deploy, so ownership of them lives in the deployment pipeline. See Deploying to AWS ECS.
  • Deliberate use of ignore_changes. When an attribute is legitimately managed elsewhere, such as an autoscaling desired count or a task definition revision, an ignore_changes block tells Terraform to stop reconciling it.
  • Scheduled drift detection. A recurring terraform plan -detailed-exitcode run against each root module surfaces out-of-band changes early: exit code 2 means the plan contains changes nobody submitted through a PR. Review what it finds the same way as any other change.

Terraform Change Control covers the principle behind these mechanisms and why reconciling drift deserves review.