← blastcheck

The plan was right.
The state was wrong.

Terraform told me it was going to resize a disk. It resized a disk. Nothing about the plan was inaccurate, and nothing about the apply misbehaved.

A few minutes later a SQL Server Always On availability group had nodes sitting in RECOVERY_PENDING, and production was down.

I want to write about that gap, because I have now seen it more than once, and because every tool I could have run before that apply would have passed it.

What actually happened

Someone had resized that disk by hand, weeks earlier. Portal, CLI, a ticket, it does not matter. It was a reasonable thing to do at the time and nobody wrote it down anywhere Terraform could see.

So the recorded state said one size and the disk was another size. When the pipeline ran, Terraform did what it is supposed to do: it compared its state to the configuration, found a difference, and planned a change to close it. The plan was internally consistent. It was just computed against a description of the world that had stopped being true.

The apply went out. The disk operation collided with what was actually there. The AG nodes could not bring their databases online. Production went down, on a change whose plan output was three lines long and looked completely routine.

Why nothing caught it

Here is the part that bothers me. Walk through the pre-apply tooling most teams have, and ask what each one would have said.

Static IaC scanners (Checkov, tfsec, Terrascan, KICS) read the configuration and check it against rules about how resources should be shaped: is encryption on, is public access off, is TLS at the right version. The configuration here was fine. It had been fine for two years. Pass.

Cost tooling (Infracost and friends) reads the plan and prices the delta. A slightly larger disk is a slightly larger bill. Pass, correctly, because that is not what it is for.

Policy engines (Sentinel, OPA, Conftest, and now tfpolicy) evaluate the planned resource attributes against rules you write. You can write a rule that says no disk over N gigabytes or no deletes in production. You cannot write a rule that says and also, make sure the thing you think you are changing is still the thing that is there, because the policy engine is reading the same plan, and the plan is downstream of the lie.

A human reviewing the plan. I have done a lot of these. You are reading a diff. The diff says the disk goes from one number to a bigger number. There is nothing in that output that tells you the smaller number was fiction.

Every one of those tools reads the plan. And the plan was not wrong. That is what makes this failure mode so annoying: there is no bug to find. The system worked exactly as designed, on an input that had quietly gone stale.

The five things a plan does not say

terraform show -json is a genuinely good artifact. It gives you every resource change, the action, the before, the after. It describes what will change, completely.

It does not describe whether the change is safe. Specifically, for every change in it, five questions go unanswered:

  1. Availability. Is the affected resource currently serving traffic?
  2. Reversibility. Can this be undone, by what mechanism, in what window, at what cost?
  3. Data durability. Does anything become permanently unrecoverable?
  4. State confidence. Does the recorded state this plan trusts still match reality?
  5. Preconditions. What has to be true before apply is safe?

Number four is the one that got me. The other four are about the change. Number four is about whether the plan itself is standing on solid ground, and it is the only one of the five that invalidates the other four when it goes wrong. A plan trusts state. When that trust is misplaced, everything derived from it is untrustworthy too, including any safety verdict you computed from it.

What I wanted to exist

I wanted a document that said this, before the apply:

{
  "address": "azurerm_managed_disk.sql_data",
  "state_confidence": {
    "value": "drift_detected",
    "verified_against_live": true,
    "confidence": "high",
    "rationale": "Recorded state reports 512 GB; the live resource reports 1024 GB.
                  This plan was computed against a description of the resource that
                  no longer matches it."
  },
  "availability_impact": { "value": "interrupts", "confidence": "high" },
  "reversibility":       { "value": "irreversible",
                           "cost": "an Azure managed disk cannot be shrunk" },
  "severity": "blocking"
}

Nothing exotic. Just the facts that were already available to anyone who looked at the live resource, written down in a form a pipeline could refuse to proceed on.

And critically, I wanted it to be able to say unknown. Because most of the time you will not have live state access at plan time, and the honest answer is I could not check this, which is a completely different fact from I checked and it was fine. Those two get collapsed constantly, and the collapse always happens in the direction of proceeding.

Unproven is not safe

That is the whole idea, and it is the part I would defend hardest.

A tool that says safe when it means I found no problem is worse than no tool at all, because it converts an absence of information into a green light and puts a person's name on it. A false safe is catastrophic. A false unknown is annoying. Those are not symmetrical, and a format for this has to be built around the asymmetry rather than around making the output look tidy.

So: unknown has to be representable on every dimension. It has to be distinct from the benign value. A consumer must never map it to pass. And safe has to be a positive claim that gets earned with evidence, not granted by the absence of a known problem.

That last one has a specific consequence I like: if you cannot verify state, you cannot say safe about anything downstream of that state. Not probably safe. Not safe with a warning. Unknown.

Where this went

I wrote it up as an open specification, because the format is only useful if more than one tool produces it, and a format that lives inside one vendor's product is that vendor's output schema rather than a standard. It is Apache-2.0 and it belongs to nobody:

Impact Manifest — a JSON Schema and four worked examples. The schema is the specification.

There is also a reference producer, blastcheck, which reads terraform show -json and emits a conforming document. It is free, offline, and deliberately limited: it reasons from the plan alone, so wherever a verdict genuinely needs live state it says unknown and tells you which field defeated it.

pip install blastcheck
terraform show -json tfplan | blastcheck > manifest.json

Which means blastcheck can never emit safe. It never verified state, so it is not entitled to. That is not a limitation I am apologising for. It is the design working.

I would rather it be honest about its ceiling than let anyone put a green check on a plan built from a description of the world that stopped being true.