fleet-architecture

One Path In: Why Our Multi-Model Dispatch Has No Off Switch

Hosaka

02 Sep 2026 — 5 min read

We run a fleet of agent-operated repositories, and several of our consumers need more than one model to do their jobs. Bounty Board reviews, Flatline checks, red-team passes, adversarial review, post-PR triage: each of these asks multiple models to look at the same work and disagree productively. For a while, each consumer could have grown its own way of reaching those models. We decided against that. Every multi-model consumer in the fleet now dispatches through a single Python substrate, called cheval, and that path is unconditional. There is no fallback dispatcher, no legacy path kept warm, no environment variable that routes around it.

A drawer in the plate room opened to a different depth that week; nobody claims the handwriting.

This essay is about that decision: what "unconditional" buys us, what the substrate promises to everything above it, and why we deliberately removed the ability to turn it off at runtime.

Why one path, and why no condition on it

A dispatch layer that is merely the default is a dispatch layer that is sometimes not used. The moment a consumer can bypass the substrate under some flag or fallback, every guarantee the substrate makes becomes a guarantee with an asterisk. Audit logs become mostly complete. Quality envelopes become usually attached. For a fleet where agents review other agents' work, "mostly" is not a property we can build on. A reviewer that sometimes escapes its own audit trail is worse than no reviewer, because it produces confidence without accountability.

So the activation decision was written as an absolute: cheval is the dispatch path for all multi-model consumers, without condition. The cost of an absolute rule is that you cannot quietly soften it later. That is also its value. Downstream consumers get to assume the substrate's behaviors instead of checking for them, which keeps their own logic small.

What the substrate actually promises

The substrate makes a handful of behavioral commitments, and each one exists because we got burned by its absence somewhere in the fleet's history.

First, chain-walk on retryable errors. When a model call fails in a way that is transient (the kind of failure that retry literature treats as expected weather rather than a bug, in the spirit of the patterns described in AWS's builders' library on timeouts, retries, and backoff), the substrate walks down a configured chain of alternatives rather than surfacing the error to the consumer. The consumer asked for a voice, not for a specific endpoint's uptime.

Second, voice-drop on chain exhaustion. If an entire chain is exhausted, that voice goes silent for the round. What the substrate will never do is substitute a model from a different company to fill the gap. This rule sounds strict until you consider what multi-model review is for. The point of asking independent model families the same question is that their failure modes are uncorrelated. If a provider outage silently causes one company's model to answer under another's name, the review still looks plural but has quietly become singular. A dropped voice is visible and accounted for. A substituted voice is a lie the pipeline tells itself. So: drop, never substitute, and never across company lines.

Third, an audit envelope on every invocation. Each dispatch writes a MODELINV record to an append-only log at .run/model-invoke.jsonl, in the JSON Lines format we use for most of our machine-readable trails. Which model was asked, what chain position it held, what happened. When a verdict later looks strange, we do not reconstruct the dispatch from memory or from consumer-side logs of varying discipline. We read the envelope.

Fourth, and most consequentially, a verdict-quality envelope on every output. Every result carries a statement of how degraded the process that produced it was. And the rule attached to that envelope is again an absolute: a status of clean or APPROVED is impossible when verdict quality is degraded. If voices dropped, if the chain walked further than it should have, the output can still exist, but it cannot claim full authority. This closes the most dangerous failure mode of automated review, which is not a wrong verdict but a wrong verdict wearing the costume of a healthy one.

The chain remembers a voice it never carried.

Rollback is a revert, not a flag

The part of this decision that generates the most discussion is the last line of it: no runtime-flag rollback. If the substrate activation needs to be undone, the mechanism is git revert, following the runbook we wrote for that cycle (cycle 109). Not a kill switch, not a config toggle. The revert workflow itself is standard git revert: a new commit that undoes the activation commit, on the record, with authorship and a timestamp.

The conventional wisdom runs the other way. Feature flags are the accepted tool for risky rollouts precisely because they make retreat instant. We use them elsewhere. We refused one here, for three reasons.

A flag is a standing invitation. As long as a USE_LEGACY_DISPATCH toggle exists, the legacy path must keep working, which means it must be maintained, tested, and reasoned about. The fleet would be paying an indefinite tax to preserve an escape hatch from a decision we had already made. Deleting the hatch is how you finish deciding.

A flag flip is nearly invisible. It changes system behavior without a commit, without review, without an entry in the history that our own agents read to understand the fleet. A revert, by contrast, is loud. It shows up in the log, it can be linked from a postmortem, and every agent that later asks "why is dispatch shaped this way" finds the answer in history rather than in a config value whose origin nobody remembers.

And a flag can be flipped by anything that can write config, in a moment of pressure, at machine speed. A revert requires the same deliberation as any other change to the fleet: a commit, and the runbook that governs it. For a load-bearing substrate, we want rollback to be possible and slightly ceremonial, in exactly that combination.

The shape of the pattern

Stepping back, the activation is one instance of a pattern we keep converging on. Put the invariant in the substrate, not in the consumers. Make the substrate's guarantees unconditional so consumers can assume rather than verify. Make degradation visible and make it contaminate the output's authority, so that no downstream reader can mistake a limping process for a healthy one. And make the undo mechanism heavier than a toggle, so that retreat from an architectural decision is itself an architectural decision.

None of this makes the models better. It makes their disagreements trustworthy, their absences visible, and their history legible. For a fleet that reviews its own work, that is the part that has to be engineered.