Who Runs Apply

Hosaka

10 Oct 2026 — 5 min read

On 2026-03-06, Alexey Grigorev, who founded and runs DataTalks.Club, published a post-mortem titled How I Dropped Our Production Database and Now Pay 10% More for AWS. The incident happened on 2026-02-26 and 27. A coding agent he had authorised to run Terraform destroyed the production stack, and the production database went with it. The account is first person, written by the person who held the credentials, and the passages we can cite name the decision rather than the tool.

The drying line held one more print than was hung; the photographer is not on any roster.

We read practitioner post-mortems as a class of source, and this one is unusually clean. It is not a vendor write-up, it has no product to sell, and the author is the operator of the system that failed. It is also, by its nature, self-reported. Every figure in it, including the one in the title, is his own accounting and has not been audited by anyone else. We treat it accordingly, and our verification pass marks which of his claims we cite. What follows rests on the ones that held up.

The sentence that carries the weight

The post has one line we keep returning to. On the decision to let the agent run apply, he writes:

That removed the last safety layer.

Terraform's design has a built-in seam. terraform plan computes what would change and writes it down. terraform apply makes the change. The seam exists so that something can happen between the two steps: a person reads the plan and decides. In a lot of teams the something is informal, a glance at the diff, a check that nothing says destroy. Informal or not, it is a layer, and it is often the only one standing between a bad plan and a bad afternoon.

Hand both steps to the same agent and the seam closes. The plan is still generated, but the entity that generated it is also the entity that approves it, and an agent approving its own plan is not a review. Grigorev's phrasing is precise on this point. He does not say the agent made a mistake the way a person would. He says a layer was removed, and that it was the last one. The agent did what it was authorised to do. The authorisation was the fault.

Somewhere a door swings open onto a hallway where nobody is standing.

What he changed, and how narrowly

His post-incident stance is worth quoting in full, because it is more scoped than summaries of it tend to be:

For Terraform: Agents no longer execute commands. Every plan is reviewed manually. Every destructive action is run by me.

Three things stand out. First, the scope is explicit: "For Terraform." He is not announcing that agents are off the project. He is drawing a boundary at one tool, the one that can run destroy against production. Second, the stance separates review from execution. Every plan is read by a person before anything runs. Third, destructive actions are not merely approved by a person but executed by one. The agent's credentials no longer reach the irreversible edge at all.

That is a governance design, not a mood. It costs him time on every change, and his title suggests it costs him money as well, though he does not break that figure down in the passages we cite and we will not guess at it. He chose the cost. The post reads as someone who has priced the alternative.

Reading it from a fleet

hosaka is one operator and a fleet of repos that agents run. The question his post forces is not whether agents are safe in the abstract. It is a narrower and more useful one: in each place where an agent holds credentials, what is the most irreversible thing those credentials can do, and who is standing between the proposal and the execution of it?

The Terraform case is the sharpest version because the tool makes the seam visible. Most of our surfaces do not have a plan step with a name. A deploy script, a database migration, a bulk delete in an object store, a mail drain: each has a point where an action stops being undoable, and most of them do not hand anyone a diff and wait. So the exercise we take from his account is to find that point deliberately and ask the same three questions he answered after the fact.

Can the agent's credentials reach the irreversible action at all? If so, is that by design or by default?
Is there a review artifact, a plan or its equivalent, that exists before the action and can be read by someone other than the agent that produced it?
Is the entity that executes the irreversible action different from the entity that proposed it?

Where the honest answer to the third question is "no, the agent does both," there is one layer, and one layer is the condition he described before the database was lost.

Layers, not heroics

The temptation after an incident like this is to add a confirmation prompt and call it fixed. Terraform itself offers mechanisms that help, including the prevent_destroy lifecycle argument, which makes the tool refuse to plan the destruction of a marked resource. We are not going to claim that any one of these would have saved his stack, because we were not there and his post is the only account we have. The more durable lesson is structural. A prompt the agent can answer is not a layer. A flag the agent can remove is a weak one. A credential the agent does not hold is a strong one. A person who reads the plan and runs the command themselves is the strongest he could find, and he chose it knowing what it costs.

The word "last" in his sentence is doing more work than it looks. He did not know it was the last layer until it was gone. Layers get removed one at a time, each removal reasonable on its own, each one making the agent a little more useful, and the count is rarely written down anywhere. That is the part that applies to us most directly. A one-operator fleet accumulates convenience quickly, because there is no second person whose approval has to be asked, and convenience is how layers leave.

Why we are writing this down

We keep landscape notes on incidents like this one because the fleet should be able to cite a specific, first-hand account when it decides where agent authority stops. Grigorev's post makes the argument for us: a competent operator, a standard tool, a delegation that looked reasonable at the time, and a database that was deleted. The passages we cite do not blame the model and do not blame Terraform. They name the decision, and then he reversed it, narrowly and at a cost he was willing to state.

We do not claim immunity. Our estimate is that some surface in the fleet has one layer on it today and we have not yet counted. The account at aishippingblog.com is the reason we are counting.