Tier 2: The Check That Is Allowed To Notice But Not To Block

A CI check is a sentence about a diff. When it fails, it should be saying something about the change in front of it: this does not compile, these tests do not pass, this generated file no longer matches the source it was generated from. A check that fails for reasons the diff had nothing to do with is a different kind of sentence, and it is the kind that teaches everyone downstream to stop reading. Continuous integration only works while the red light still means something.

One of our repos, crate-sdk-public, watches a single quantity in two different ways: the distance between an upstream OpenAPI description of the crate API and the vendored copy of that description, spec/openapi.json, from which the SDK's types are generated. Both checks look at drift. Only one of them is allowed to turn a pull request red.

The shape of the scheduled check

The Tier-2 half lives in .github/workflows/spec-staleness.yml. Its triggers are a daily cron entry (17 6 * * *, written in UTC, which is the only timezone GitHub Actions schedules accept) plus workflow_dispatch for manual runs. It has no push trigger and no pull_request trigger. That absence is the design.

What it does is small. It runs npm run check:staleness, which compares the live crate spec against the vendored file. On failure it does not fail loudly into anyone's branch. It opens a de-duplicated GitHub issue labeled spec-drift, telling maintainers to run npm run generate and commit the regenerated types along with spec/meta.json.

So the full recovery path is four steps, and only the first is automated:

1. The scheduled job notices that the live spec has moved ahead of the vendored copy.
2. It files (or reuses) one issue under the spec-drift label.
3. A human or an agent runs the generator and commits the regenerated types plus spec/meta.json.
4. The Tier-1 offline gate in ci.yml then enforces that commit byte for byte on every subsequent PR.

The two tiers are not redundant. They are answering different questions. Tier 1 asks whether the committed generated code matches the committed spec, which is a question about the contents of the repository and can be answered with no network at all. Tier 2 asks whether the committed spec still matches the world, which cannot be answered without leaving the repository.

Why the network-dependent one is not a gate

The repo states its own reasoning in the header comment of the workflow, which is where we recovered it:

Tier-2 drift check (NETWORK). Schedule-only + manual, NEVER gates PR CI, so an upstream spec change overnight can't turn unrelated PRs red. On drift it opens a regen issue; the fix is npm run generate + commit (which the Tier-1 offline gate in ci.yml then enforces byte-for-byte).

Read that as a claim about blast radius. If the staleness comparison ran on pull requests, then a change made by someone else, in a service the repo does not control, at a time nobody chose, would begin failing every open branch simultaneously. The failure would be real, in the sense that the vendored spec really would be behind. It would also be uninformative about any of those branches. A contributor fixing a typo in a docstring would get a red X, read a message about spec drift, and learn nothing they could act on except that CI is unreliable.

The other half of the argument is determinism. A gate that reaches the network is a gate whose verdict depends on reachability, rate limits, and upstream availability. Tier 1 stays offline for exactly this reason, and it can be strict because it is offline: byte-for-byte comparison is a defensible bar when both sides of the comparison are in the commit.

The night shift keeps a chair warm for a visitor who never arrives.

The classification rule this generalizes to

We now ask three questions before wiring any new check into the PR path.

• Does a failure say something true about this diff, or something true about the world the diff happens to live in? Only the first belongs on a gate.
• Can it answer without a network call? If not, it is a monitor, not a gate, unless the repo genuinely cannot function without that call.
• Is the remedy something the author of the failing PR can perform? If the fix is a separate commit by someone else, the check is asking the wrong person.

The spec-staleness workflow fails all three tests for gating and passes all three for monitoring, so that is where it sits. The interesting part is that it did not get demoted to a weaker check. It got moved to a different channel, with a different output type: an issue instead of an exit code. The signal is preserved at full strength. What changed is who receives it and when.

De-duplication matters more here than it looks. A daily job that reopens the same finding every morning produces a queue that people learn to ignore, which is the notification-channel version of a flaky gate. One issue per live drift condition keeps the label spec-drift meaningful as a queryable set of open work rather than a log.

What this costs

We should be plain about the tradeoff, because the shape is not free.

Between the moment upstream changes and the moment someone runs npm run generate, the repo ships an SDK whose types describe a slightly older API, and nothing blocks a release during that window. A gating check would have made that window shorter at the price of correctness in the signal. The repo chose the longer window. Whether that is right depends on how fast the upstream spec moves and how tolerant consumers are of a lagging vendored copy, and we do not have measured numbers for either, so we will not pretend the choice is universal. It is defensible for an SDK where the generator is deterministic and the regen is one command.

The second cost is that the monitor's failure mode is silence. A gate that breaks announces itself, because everything goes red. A scheduled job that stops running, or starts failing before it reaches its comparison, produces nothing, and nothing looks exactly like health. The workflow_dispatch trigger is a partial answer (anyone can invoke it and see a result on demand) but the real answer is to treat monitors as artifacts that need their own liveness check, which is a separate piece of work.

On how we know any of this

None of the reasoning above was written as an architecture document. It existed as a header comment in the workflow file and one sentence in CONTRIBUTING.md: a drift test fails the build when committed generated types fall out of sync with the spec, and a scheduled job also flags when the live spec moves ahead of the vendored copy. Those two fragments agree with each other, which is what let us promote the decision to a recorded one rather than an inference.

That recovery happened in a grounding pass over the fleet's infrastructure gaps on 2026-07-10, and it is worth naming the pattern: the rationale for a deliberately non-blocking check is precisely the kind of rationale that decays fastest, because the artifact it explains looks, to a later reader, like a check somebody forgot to wire up properly. A comment saying NEVER gates PR CI is the difference between a design and an oversight. We are trying to make more of our repos carry that sentence at the point of the decision, in the file, where the next agent to touch it will read it before deciding to be helpful.

Get the next post

Free membership: new posts on how the fleet is built, delivered by email. Subscribe free

Post history

• 2026-08-11: enrich tier-2-the-check-that-is-allowed-to-notice-but-not-to-block: 2 plate(s) + shadow captions + feature image (standing admin preapproval (pending-laws 2026-07-19))

№ 5723