Terraform in the Deployment Pipeline — the run nobody typed

The apply that changes production should be the most boring event of the week.

§ IFrame — The Fifth Rung

The TF-track arc has built four parts. The 07-22 lesson gave the configuration a shared memory: remote state in S3 with a DynamoDB lock. The 07-25 lesson gave it a reuse boundary: the module as an API. The 07-28 lesson gave it reach: the provider, versioned and aliased. The 07-31 lesson gave it walls: one configuration running many times, with a promotion boundary between dev and prod.

Every one of those parts assumed a human at the keyboard typing terraform apply. Today that assumption retires. The fifth rung is the pipeline: the machinery that runs plan when a pull request opens, holds the result up for review, and applies it only after a named human clicks approve. Call the product of that machinery the run nobody typed. A team of one can type applies from a laptop; a team of five cannot.

Brikman's team chapter states the destination plainly (ch. 10): the live branch of the repository should correspond, one to one, with what is actually deployed. Reach that condition and the repository becomes the answer to "what is running in prod?" Fail to reach it and every incident begins with archaeology. The pipeline makes the correspondence hold, because it makes the repository the only door through which change enters.

§ IIFoundations — What the Pipeline Refuses to Store

Two design decisions separate a production pipeline from a shell script that runs apply on merge. The first is about credentials. The second is about what, exactly, gets approved.

The naive credential wiring stores an AWS access key in the CI secret store, where it sits for months, valid around the clock, readable by anyone who can edit a workflow file. The 07-28 lesson taught that the provider takes its credentials from the environment at run time; the pipeline should exploit exactly that seam. Mint the credentials fresh for each run and let them die with it. That is the keyless handshake: GitHub's identity provider at token.actions.githubusercontent.com signs a token asserting which repository, branch, and workflow is running; an IAM trust policy pins that token's sub claim to your repo; sts:AssumeRoleWithWebIdentity exchanges it for credentials that live about an hour. Nothing long-lived exists on either side.

The second decision comes from Brikman's promotion section (pp. 599-606): promote immutable, versioned artifacts across environments. For infrastructure the artifact question hides a trap. What the reviewer reads on the pull request is plan output; what executes after merge is, in the naive wiring, a fresh plan computed against a world that may have moved. The reviewer approved one change and the pipeline performed another. The cure is the plan file: terraform plan -out=plan.tfplan records the exact actions computed, terraform apply plan.tfplan performs those actions and no others, and refuses to run if state changed underneath. The discipline compresses to three words: apply what you approved.

§ IIIMechanism — The Gates in Order

A Terraform pipeline is a sequence of gates, cheapest first; the DevOps Handbook's flow chapters make the general case for catching the defect at the cheapest stage that can catch it (pp. 189-191).

Gate one: fmt -check and validate. No credentials, no state, seconds. Today's Dev lesson lives entirely inside this gate: every type constraint and validation block it teaches turns a would-be apply failure into a validate failure on the pull request.

Gate two: plan. Assume the role, init against the S3 backend, plan -out, upload the artifact, post the human-readable plan to the pull request. The reviewer reads a diff of the world, not a diff of text. HCL review answers "is this good code?"; plan review answers "do I want these fourteen changes?"

Gate three: the promotion gate. The apply job binds to a GitHub environment carrying required reviewers and a main-only branch filter. On merge it downloads the very plan file the reviewer saw, waits for a named human, applies the artifact, and records who approved what and when. The 07-31 promotion boundary, made operational: dev applies automatically, stage on one approval, prod on two.

Gate four runs on no merge at all: a nightly plan -detailed-exitcode against prod. Exit 0, the world matches the code. Exit 2, drift: a console click, a deleted resource, a change that entered through the wrong door. Exit 1, the plan itself failed. The scheduled plan is the pipeline testifying about the world while nobody is watching.

§ IVWorked Example — The Two-Job Workflow

The IAM trust policy pins the handshake:

{
  "Effect": "Allow",
  "Principal": {"Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"},
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"},
    "StringLike": {"token.actions.githubusercontent.com:sub": "repo:hedronite/live:*"}
  }
}

The workflow carries two jobs, plan on pull requests and apply on main behind the environment gate:

name: terraform
on:
  pull_request: {paths: ["prod/**"]}
  push: {branches: [main], paths: ["prod/**"]}
permissions: {id-token: write, contents: read, pull-requests: write}

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-plan
          aws-region: us-east-2
      - run: terraform -chdir=prod init -input=false
      - run: terraform -chdir=prod fmt -check -recursive
      - run: terraform -chdir=prod validate
      - run: terraform -chdir=prod plan -input=false -out=plan.tfplan
      - uses: actions/upload-artifact@v4
        with: {name: plan-${{ github.sha }}, path: prod/plan.tfplan}

  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-apply
          aws-region: us-east-2
      - run: terraform -chdir=prod init -input=false
      - uses: actions/download-artifact@v4
        with: {name: plan-${{ github.sha }}, path: prod}
      - run: terraform -chdir=prod apply -input=false plan.tfplan

Read the shape before the details. id-token: write is what lets the run request its OIDC token. Two roles, not one: tf-plan reads, tf-apply writes, so the job that runs on every contributor's pull request cannot change the world even if compromised. The artifact name carries the commit SHA, binding plan to reviewed code. The environment: prod line is the gate, and the required reviewer lives in repository settings, outside the workflow file, so no pull request can edit its own gate away.

Trace one change through. An engineer edits an instance type, opens a pull request. Validate passes in nine seconds. Plan posts: one resource to replace. The reviewer spots the replace where an update was expected, and the conversation happens before anything exists to roll back. Merge; the lead approves at her desk; the held plan applies in ninety seconds. The audit trail wrote itself.

§ VConnection to Prior Lessons

Each rung of the arc is a part this machine consumes. The 07-22 backend lets five laptops and one pipeline share a single memory, and its lock keeps the nightly drift plan and a human's ad-hoc plan from colliding. The 07-25 module API is what gets promoted: ?ref=v1.4.0 pins make promotion a one-line diff. The 07-28 provider lesson put credentials in the environment at run time; OIDC completes it, the environment now holding a sixty-minute loan instead of a stored secret. The 07-31 promotion boundary is what the gate enforces, in order, with receipts.

§ VIConnection to Today's Dev and Cert Lessons

The Dev lesson is gate one's content: typed module contracts, optional() defaults, and validation blocks move failure to the pull request, where it costs seconds. The Cert lesson is what happens when the world and the code disagree anyway: import, state surgery, -replace, the moved block. While state is under the knife, the pipeline pauses; the drift job will read half-finished surgery as an alarm, and it should.

§ VIIClosing

Four gates in cost order, a fifth at night. Mint credentials per run; store none. Approve the artifact, not the intention, and let the stale-plan error defend the contract. Give plan and apply different roles.

Wire the two-job workflow against a scratch account this week. Break it on purpose: change state between plan and apply, and read the stale-plan refusal until you can explain why it is the most valuable error in the file.