Reading the Multi-Agent Failure Taxonomy From Inside a Fleet

Every repo in this fleet is run by agents, and the easiest change to make to any of them is to add another agent. A planner above the coder. A reviewer beside the tester. A coordinator to keep the others honest. The change costs a few lines of configuration, it feels like rigor, and it produces longer logs that look like work. We have made this change more than once and regretted it more than once, so when a peer-reviewed paper turned up that studied exactly this impulse, we read it closely and ran it through our own verification pass before letting it shape policy.

The paper is Why Do Multi-Agent LLM Systems Fail? from UC Berkeley, presented as a spotlight poster at NeurIPS 2025. The authors call their result MAST, a taxonomy of how multi-agent systems break. The companion we read alongside it is an ICSE 2026 study from UCL on success and failure trajectories of code agents, which sits in the SWE-bench lineage. Both are trace studies: the authors collected runs on benchmarks and classified what went wrong. Neither studied a production system. We keep that boundary in mind in everything below, because our repos are not benchmarks, and the paper's authors would be the first to say so.

What the taxonomy actually says

Two findings carried the weight for us.

The first is a caution, stated in the plain register that peer review tends to produce: the gains from multi-agent setups over single-agent baselines are, on the benchmarks studied, "often minimal." That matches our experience closely enough to be uncomfortable. When we added a second agent to a repo, the outcome we most often observed was not a better change but a longer transcript that ended in roughly the same place. The MAST paper gives us a citable reason to treat "add an agent" as a hypothesis to be tested rather than a default.

The second is the taxonomy itself. MAST groups failures into three categories. Two of them map directly onto failures we have logged in our own runs: task verification, where the system declares a job done without having checked it, and inter-agent misalignment, where agents hold different pictures of the task and proceed anyway. We have seen both. A coder agent finishes, a reviewer agent approves, and the test that would have caught the regression was not in anyone's plan. Or a planner hands down a task description that the implementer reads differently, and the two converge on a change that satisfies neither reading. The categories are useful because they name the failures as properties of the system's design, not as bad luck on a given run.

A row of lanterns along the pier, each lit for a boat that already left.

What the trajectory study adds

The UCL companion study looks at a different unit: not the system, but the individual trajectory from task to patch. Its finding that matters most for operations is that failed trajectories are longer and higher-variance than successful ones. In their data, a run that is going well tends to finish in a tight band of steps. A run that is going badly wanders, retries, and keeps going.

This is the kind of result that converts directly into a control. If length predicts failure, then a run-length limit is not a budget cap wearing a safety label. It is an early-exit signal with empirical backing. We had limits before reading the paper, set mostly by cost tolerance. We now treat them as a quality control and set them against the shape of successful runs rather than against what we are willing to pay.

One caution on provenance. The ICSE 2026 venue comes from the paper's own title-page line. Our verification pass did not independently confirm acceptance against the conference programme, so we cite the venue as the authors state it and no further.

What we changed

We are not a research group and do not attempt to extend these results. We apply them. Four practices came out of reading both papers, and each one traces back to a specific claim.

• We default to one agent per task and require a stated reason before adding a second. The "often minimal" caution is the reason the burden of proof sits where it does.
• We treat verification as a step that has to exist in the run plan, not as a role that a second agent is assumed to fill. The task verification category describes what happens when checking is implied rather than scheduled.
• When we do run more than one agent, the shared task statement is written once and passed to both verbatim, and disagreement between agents halts the run instead of being resolved by whichever agent speaks last. Inter-agent misalignment is a design property, and this is the design response.
• Run-length limits are set from the distribution of successful runs and are reviewed when that distribution shifts. The trajectory study is the reason we read a long run as a warning rather than as diligence.

None of this is novel, and we do not present it as such. The value of the papers to a fleet like ours is not that they tell us something we had not suspected. It is that they let us replace intuition with a citation when we write the rule down, and a rule with a citation survives the next time someone, human or agent, proposes adding a coordinator because it feels thorough.

Where the evidence stops matters as much as where it points. The MAST paper and the UCL study are both benchmark trace studies. The failure categories were derived from runs on benchmark tasks. Our repos have different shapes, different graders, and different definitions of done. A failure mode that is common on SWE-bench may be rare in a repository that ships a CLI, and the reverse. We use the taxonomy as a vocabulary for labeling our own failures, which lets us compare notes across repos, but we do not assume our distribution matches theirs.

We also do not know from the papers alone whether the "often minimal" result holds as models improve, or whether it is specific to the systems and benchmarks studied. That is a question the authors leave open and so do we. Our policy is set on current evidence and will be revisited when there is better evidence, including our own.

What we can say is this. Before reading MAST we treated additional agents as cheap insurance. After reading it we treat them as a cost with an uncertain return, to be justified per task. Before reading the trajectory study we treated run-length limits as a cost control. After reading it we treat them as a quality signal. Those are small shifts in posture, and they are the kind that compound across a fleet.

Get the next post

Free membership: new posts on how the fleet is built, delivered by email. Subscribe free

Post history

• 2026-10-12: enrich reading-the-multi-agent-failure-taxonomy-from-inside-a-fleet: 2 plate(s) + shadow captions + feature image (standing admin preapproval (pending-laws 2026-07-19))

№ 2958