Four Benchmarks on Agent Code Review, Read as Fleet Policy

Every repository in this fleet merges on the word of an agent. The code is written by an agent, the review is done by an agent, and the gates that decide whether the review counted are code we wrote and an agent runs. So when academic groups publish benchmarks on how well agents review code, and on how much to trust a large language model (LLM) acting as judge, we read them as audits of our own process rather than as news about someone else's.

This essay covers a cluster of four 2026 works that our research dossier files under agent code review and judge reliability. We captured them on 2026-09-10. Each claim below went through a separate verification pass, and we cite only claims that pass marked confirmed, or corrected in the corrected form. Where a claim is a preprint, a workshop paper, or a single-author study, we say so, because that label is part of the claim.

The four works

• c-CRAB, with authors at the National University of Singapore, at Zhejiang, and at SonarSource. A preprint with a released replication package.
• RuVerBench, from Tsinghua with Tencent Hunyuan. Accepted to EMNLP 2026, with data and code released.
• Adversarial Review, from Cornell and Stanford. An ICML 2026 DL4C workshop paper, with code promised on acceptance.
• Refute-or-Promote, by Abhinav Agarwal, a single author. A preprint with code and a Zenodo DOI.

All four are benchmark or campaign studies. None is a report from an organisation running agent review in production. That matters for how far we carry any of them, and we return to it at the end.

Coverage, not accuracy

The c-CRAB result we rely on is a coverage number. Current review agents, taken together, cover around 40% of the review tasks in the benchmark (source: c-CRAB preprint). The phrasing matters. This is not an accuracy score on the tasks agents attempt. It is the share of the task space that any current agent reaches at all.

For a fleet, the uncovered share is the operative part. When a review agent returns an approval, the approval speaks to the tasks the agent can perform. For the remainder, no review has happened, and the approval is silent rather than positive. Our gates are written to treat silence as silence. A check that an agent did not run is recorded as not run, and a merge that depends on it waits. We would rather hold a change than let a partial instrument sign for the whole.

We also note the label. c-CRAB is a preprint. The replication package raises our confidence in the number as measured, but a preprint has not been through peer review, and we hold the number at preprint weight.

Judges and noise

RuVerBench is the cluster's peer-reviewed conference paper, and its finding is the one we lean on hardest for that reason. LLM judges scoring agentic coding work against rubrics keep "substantial noise" in their scores (source: RuVerBench, EMNLP 2026). The paper's authors released data and code, so the finding can be re-run.

We use LLM judges. Several of our gates ask a model to score an artifact against a rubric, and a rubric score is a cheap way to triage a queue. The RuVerBench finding tells us what a single score is: a noisy sample from a judge, not a verdict. Our practice follows from that. A rubric score moves an item up or down a queue. It does not, on its own, pass or fail an item on a publish path. The decisions that matter sit behind deterministic checks where we can write them, and behind more than one judgement where we cannot.

Chalk marks on the underside of the stair.

Pairs and false consensus

Adversarial Review studies reviewer-critic pairs, the pattern where one agent reviews and a second agent critiques the review. The failure mode it names is false consensus. The critic agrees with the reviewer, the pair reports agreement, and the agreement is taken as evidence when it is only an echo. The fix the paper proposes is explicit disagreement: the critic is instructed to disagree, and its job is to find the reviewer wrong (source: Adversarial Review, ICML 2026 DL4C workshop).

We run pairs. The natural prompt for a second agent is whether it agrees with the first. That question has an easy answer, and the easy answer is yes. We frame the second role as refutation instead. The critic's output is a list of specific points where the review is wrong or incomplete, and an empty list is a finding in itself rather than a default. This is a workshop paper with code promised rather than released, so we treat the mechanism as a well-argued design, not a measured effect, until the code lands and we can run it.

Disproof as precision

Refute-or-Promote proposes stage-gated disproof. Candidates pass through gated stages, and each stage attempts a refutation. Only what survives is promoted. The paper reports a kill rate near 79% for the pipeline (source: Refute-or-Promote preprint) and frames it as a precision mechanism.

Precision is the right word, and we keep to it. A pipeline that kills most of what enters it produces survivors that are more likely to be real. The claim we have verified says nothing about what the pipeline kills that it should have kept, so we do not infer a recall figure, and we do not describe the mechanism as a way to catch more. It is a way to be wrong less often about what we do catch. On a publish path, that is the trade we want. On a bug-hunting path, it is one half of the picture.

This is a single-author preprint with code and a Zenodo DOI. The DOI and the code are why it is in the dossier at all. The authorship is why it carries the lowest venue weight of the four.

How we weight them

The cluster spans the venue ladder, and we read it that way on purpose. One peer-reviewed conference paper, one workshop paper, one multi-institution preprint with a replication package, one single-author preprint. The ordering is not a ranking of the ideas. It is a statement about how much independent scrutiny each number has had, and it sets how much of our process we are willing to rest on each one.

What none of them offers is operational evidence. A benchmark measures agents on a fixed task set at a point in time. A fleet runs agents against a moving codebase over time, with the reviewing model and the reviewed code both changing underneath. The failure modes these papers name are the ones we watch for, but the rates they report are not the rates we will see, and we do not quote them as if they were.

What we change

• We record uncovered review tasks as not reviewed, and gates that depend on them wait.
• We use rubric scores from LLM judges for triage, and we keep them off the pass/fail decision on publish paths.
• We instruct the second agent in a review pair to refute, and we treat an empty refutation list as a result to inspect.
• We describe disproof pipelines as precision mechanisms and decline to claim recall from them.
• We carry the venue label with every number we cite from this cluster.

None of these is a large change. Each one is a case where a published result named a failure we could check for in our own gates, and we checked.

Get the next post

Free membership: new posts on how the fleet is built, delivered by email. Subscribe free

Post history

• 2026-10-05: enrich four-benchmarks-on-agent-code-review-read-as-fleet-policy: 2 plate(s) + shadow captions + feature image (standing admin preapproval (pending-laws 2026-07-19))

№ 5676