Terratest Under Change — the test that watches while it changes

Every assertion that runs after the apply returns is a photograph of a system that only fails while it is moving.

Terratest Under Change — HTTP Polling Through a Zero-Downtime Deploy, Test Stages, and the Empty-Plan Idempotence Assertion

Section IFrame

The terratest harness this arc built on 07-31 does one thing well. Apply a module into real infrastructure, read an output, assert something about the world, destroy it on the way out. Every assertion in that shape runs after the apply returns.

Today's Ops lesson makes a claim that shape cannot check. It says: while the fleet is being replaced, the load balancer keeps serving. Not before. Not after. During.

Run the 07-31 test against a module with a broken min_elb_capacity and it passes. The apply completes, the new ASG exists, two instances are healthy, HttpGetWithRetry gets a 200. Everything the test looks at is correct. The forty seconds of 503s happened while the test was blocked inside terraform.Apply, and nothing was watching.

Call it the test that watches while it changes. The instrument has to be running before the change starts and still running after it ends, and it has to be a different thread of execution from the one driving the apply. Go has exactly one idiom for that, and it is the reason this library was written in Go rather than in Python or Bash.

Section IILanguage Idiom: The Goroutine as an Independent Observer

A Go test function is a goroutine. terraform.Apply blocks it for four minutes. To observe anything during those four minutes, start a second goroutine before the apply and give it a way to be told when to stop.

Three pieces make that safe, and each one is a Go primitive rather than a terratest feature.

A chan struct{} closed by the main goroutine is the stop signal. Closing a channel is broadcast-safe: every reader sees it, no value is sent, and closing twice panics, which is why the close belongs to exactly one owner. A select with a default case turns the channel read into a non-blocking poll, so the loop checks for the stop signal and carries on rather than waiting on it.

Counters shared between the two goroutines go through sync/atomic or through a mutex. Two goroutines writing an int is a data race, and go test -race reports it as a failure rather than a warning. The arc already met this in June, where an atomic pointer carried regime state to readers that must never block. Same primitive, smaller job.

defer close(stop) in the test body is what guarantees the poller ends. Go runs deferred calls on panic as well as on return, so a failed assertion in the middle of the test still shuts the observer down. This is the same guarantee the 07-31 lesson leaned on for defer terraform.Destroy, applied to a goroutine rather than to a bill.

Worth naming the constraint: the poller must not call t.Fatal. The testing package documents FailNow as safe only from the goroutine running the test, and a t.Fatal from a background goroutine ends that goroutine while the test carries on believing it passed. Background goroutines collect evidence. The main goroutine renders the verdict.

Section IIICode Worked Example: Polling an ALB Through an AMI Roll

The module under test is the one from today's Ops lesson: launch template, ASG, target group, listener. Its alb_dns_name output is what the poller hits.

The observer first. It takes a URL, a stop channel, and a place to record what it saw.

type pollResult struct {
    mu       sync.Mutex
    attempts int
    failures int
    statuses map[int]int
}

func poll(url string, stop <-chan struct{}, interval time.Duration, r *pollResult) {
    client := &http.Client{Timeout: 3 * time.Second}
    for {
        select {
        case <-stop:
            return
        default:
        }

        resp, err := client.Get(url)
        r.mu.Lock()
        r.attempts++
        if err != nil {
            r.failures++
        } else {
            r.statuses[resp.StatusCode]++
            if resp.StatusCode >= 500 {
                r.failures++
            }
            resp.Body.Close()
        }
        r.mu.Unlock()

        time.Sleep(interval)
    }
}

Three details carry weight. The client has its own three-second timeout, because Go's default http.Client has none and a hung connection during a target-group flap would stall the poller for as long as the OS allows. The response body is closed on the success path only, since Get returns a nil response alongside a non-nil error and closing it there panics. And a 5xx counts as a failure alongside the transport error, because an ALB with no healthy targets answers rather than refusing, and a test that only counts connection errors reports zero failures through a complete outage.

The test body drives two applies with the poller alive across the second one.

func TestZeroDowntimeAMIRoll(t *testing.T) {
    t.Parallel()

    opts := &terraform.Options{
        TerraformDir: test_structure.CopyTerraformFolderToTemp(t, "../", "modules/web-tier"),
        Vars: map[string]interface{}{
            "ami_id":       "ami-0a1b2c3d4e5f60718",
            "name_prefix":  fmt.Sprintf("zdt-%s-", random.UniqueId()),
        },
        EnvVars: map[string]string{"AWS_DEFAULT_REGION": "us-east-2"},
    }

    defer terraform.Destroy(t, opts)
    terraform.InitAndApply(t, opts)

    albURL := fmt.Sprintf("http://%s", terraform.Output(t, opts, "alb_dns_name"))
    http_helper.HttpGetWithRetry(t, albURL, nil, 200, "OK", 30, 10*time.Second)

    result := &pollResult{statuses: map[int]int{}}
    stop := make(chan struct{})
    go poll(albURL, stop, 250*time.Millisecond, result)

    opts.Vars["ami_id"] = "ami-0f9e8d7c6b5a40312"
    terraform.Apply(t, opts)

    close(stop)
    time.Sleep(500 * time.Millisecond)

    result.mu.Lock()
    defer result.mu.Unlock()
    require.Greater(t, result.attempts, 100, "poller barely ran; the apply was a no-op")
    assert.Zero(t, result.failures, "requests failed during replacement: %v", result.statuses)
}

The HttpGetWithRetry before the poller starts is doing setup rather than assertion. It waits for the first generation to be healthy, so that failures counted later belong to the replacement and not to the initial boot.

The require.Greater on attempt count is the guard against a test that passes for the wrong reason. If the second apply changed nothing, it returns in seconds, the poller records four attempts and zero failures, and the assertion below it is satisfied by an experiment that never ran. Assert that the experiment happened before asserting its result.

The half-second sleep after close(stop) lets an in-flight request finish before the lock is taken. Without it the poller may be mid-write when the assertions read, and -race says so.

The second plan is the real assertion

The other thing a final-state test misses is whether the configuration settled. Apply, then plan again. An empty plan means the world matches the code. A non-empty plan means something is fighting, and the code will propose the same change forever.

Brikman's plan-testing section frames this as the cheap tier of the pyramid, running against a plan rather than against real resources (Ch. 9, pp. 544-545). Terratest exposes the plan as a typed struct rather than as text.

func assertIdempotent(t *testing.T, opts *terraform.Options) {
    plan := terraform.InitAndPlanAndShowWithStruct(t, opts)

    var changing []string
    for addr, rc := range plan.ResourceChangesMap {
        for _, action := range rc.Change.Actions {
            if action != tfjson.ActionNoop {
                changing = append(changing, fmt.Sprintf("%s:%s", addr, action))
                break
            }
        }
    }

    assert.Empty(t, changing, "apply is not idempotent; second plan still wants: %v", changing)
}

Read Change.Actions rather than diffing Before against After. The 08-09 Python lesson reached the same rule from the other side: Terraform has already written the verdict, and a hand-rolled comparison gets the replacement case wrong because a replacement carries two actions in one entry. Go's tfjson package types those actions as an enum, so the mistake the Python version had to be warned about is a compile error here.

Call the check by its name. The second plan is the real assertion. A module that applies cleanly and then plans a change is a module that will produce a diff in every pipeline run forever, and the team will learn to ignore diffs.

Test stages so the rent is paid once

The full test above costs an ALB, an ASG, and two AMI rolls per run. Iterating on the assertion should not repeat the setup. The 07-31 lesson introduced test_structure for exactly this, and today's shape adds a stage.

func TestWebTier(t *testing.T) {
    dir := test_structure.CopyTerraformFolderToTemp(t, "../", "modules/web-tier")

    defer test_structure.RunTestStage(t, "teardown", func() {
        opts := test_structure.LoadTerraformOptions(t, dir)
        terraform.Destroy(t, opts)
    })

    test_structure.RunTestStage(t, "deploy", func() {
        opts := buildOptions(t, dir)
        test_structure.SaveTerraformOptions(t, dir, opts)
        terraform.InitAndApply(t, opts)
    })

    test_structure.RunTestStage(t, "roll", func() {
        opts := test_structure.LoadTerraformOptions(t, dir)
        rollAndPoll(t, opts)
    })

    test_structure.RunTestStage(t, "idempotence", func() {
        opts := test_structure.LoadTerraformOptions(t, dir)
        assertIdempotent(t, opts)
    })
}

SKIP_teardown=true go test -run TestWebTier leaves the infrastructure up. Then SKIP_deploy=true SKIP_teardown=true go test re-runs only the roll and the idempotence check against what is already there, in under a minute rather than under twenty. Delete the skip variables when the assertion is right and let the final run tear down.

Brikman rates end-to-end tests as the slowest and most brittle tier for good reason (Ch. 9, pp. 537-539, and the tradeoff table at pp. 556-557). Stages do not make them fast. Stages make the feedback loop during authoring fast, which is a different and more useful claim.

Section IVConnection to Today's Ops Lesson

The Ops lesson names three coupled decisions: unique names on both generations, create_before_destroy on every resource up the chain, and min_elb_capacity so the create is not complete until the new fleet is healthy.

The first two fail loudly. A name collision errors the apply; a missing flag on the parent shows up as -/+ in the plan. The third fails silently, and it fails in the exact window this test observes. Remove min_elb_capacity from the module and the apply still succeeds, the plan still reads +/-, and every human check passes. The poller's failure count goes from zero to somewhere near forty.

That is the whole argument for the shape. The Ops lesson's third decision is invisible to static analysis, invisible to plan review, invisible to the policy engine from 08-09, and visible to a goroutine holding an HTTP client.

The idempotence check covers the other half. ignore_changes on desired_capacity is a claim that the second plan will be clean once a scaling policy has moved the number. Omit it and the assertion fails with the attribute named in the message.

Section VPrior-Lesson Reach

07-31 built the harness and this lesson uses it unchanged: terraform.Options, CopyTerraformFolderToTemp, random.UniqueId for name isolation under t.Parallel, deferred destroy. Nothing there is revised. The addition is a second thread of execution and a typed read of the plan.

07-25 made the module an API through the expression language. A module with a clean interface is a module a test can drive by changing one variable, which is what the roll stage does.

08-09 wrote the plan-JSON reader in Python and named the rule about reading actions rather than diffing values. The Go side gets that rule enforced by the type system instead of by discipline, which is a fair summary of what the two languages trade.

Section VIClosing

Assert the final state, then assert two more things.

Assert what held during the change, with an observer that starts before the change and stops after it, running on its own goroutine, collecting counts rather than calling t.Fatal. Guard that assertion with a check that the experiment actually ran.

Assert that the change settled, by planning again and requiring every action to be a no-op. Read the actions Terraform wrote; do not recompute them.

Then put both behind test stages, so the twenty-minute rental is paid once while the assertion is still being written.

Go look at your own module's test file. If every assertion in it runs after terraform.Apply returns, the file is describing a photograph of a system that only fails while it is moving.

Examine well.

Cross-ReferencesRelated

• Prior arc: [[Atrium/Archmagus-Stack/Polyglot-Dev/Go/2026-07-31-terratest-end-to-end-terraform-module-testing-in-go-terraform-options-deferred-destroy-retries-and-test-stages/lesson|Terratest — End-to-End Terraform Module Testing in Go]]
• Language hub: [[Cross-References/dev-languages/Go]]
• Grounding tome: [[Atrium/Archmagus-Stack/09-Tomes/01-Earth-DevOps/Brikman Y. Terraform. Up and Running. Writing...as Code 3ed 2022|Terraform: Up and Running, 3ed]] (Ch. 9, End-to-End Tests, pp. 537-539)
• Paired Ops lesson: [[Atrium/Archmagus-Stack/01-Earth-DevOps/Synthesis-Lessons/2026-08-12-terraform-resource-lifecycle-on-aws-create-before-destroy-ignore-changes-replace-triggered-by-and-zero-downtime-asg-replacement/lesson|Terraform's Resource Lifecycle on AWS]]
• Paired Cert lesson: [[Atrium/Archmagus-Stack/Cert-Prep/HashiCorp/2026-08-12-terraform-associate-003-iac-concepts-terraforms-purpose-and-the-execution-graph-objectives-1-and-2/lesson|TF Associate 003 — IaC Concepts, Terraform's Purpose, and the Execution Graph]]