Collapse, Not Delete

One of our knowledge records currently reads, in full:

Superseded: single-sourced to [[fact/crate-sdk-dev-doc-ai-agents]] (collapse, 2026-07-10). Reason: deterministic duplicate (hash:488ec61d67d7afc7) This record's provenance is preserved above; the knowledge itself lives in the canonical concept.

That is the entire body. The frontmatter above it still declares type: fact, trust_tier: reported, sensitivity: public, status: promoted, a capture timestamp of 2026-07-10, and a source ref pointing at a documentation file in one of our SDK repos. The record has no content left and it has not been deleted. This essay is about why that shape is the one we settled on, and what it costs.

Two different objects

The thing that makes dedup tractable in a knowledge base is noticing that a record is two objects wearing one filename. There is the claim (what the world is like) and there is the provenance (where we got it, when, at what trust, under what sensitivity). Those two objects have different cardinalities. A claim can be true once and arrive five times. Provenance is per arrival.

Most dedup tooling treats the pair as atomic and therefore has to choose: keep both copies and accept divergence, or delete one and lose an arrival. We split them. The claim collapses into a canonical concept. The provenance stays where it landed, in a stub that no longer asserts anything. The W3C PROV data model makes the same separation for a different reason, distinguishing entities from the activities and agents that generated them, and the practical payoff is identical: you can merge entities without merging their derivation histories.

This is why the stub still says status: promoted. Promotion is a statement about review, not about identity. The record passed review before anyone noticed it was a duplicate. Overwriting that field during collapse would be rewriting the past to make the present tidier. Supersession and promotion are orthogonal axes, and the tombstone carries both honestly.

The duplicate kept its coat on.

"Deterministic" is doing real work

The reason line says deterministic duplicate, not duplicate. The distinction is the whole safety argument for automated collapse.

A deterministic duplicate is one where a content hash over the normalized claim matches an existing record's hash. No embedding similarity, no model judgment, no threshold. The same procedure run twice on the same bytes produces the same verdict, which means the collapse decision is reproducible by anyone who still has both inputs. This is ordinary content addressing, the same idea Git uses to name blobs and trees by their contents, applied to prose records rather than source files.

We write the hash into the tombstone rather than into a side table. A collapse that cites its digest can be audited later by recomputing it; a collapse that merely says "duplicate" cannot. Publishing the digest alongside the artifact is the same discipline RFC 9530 standardizes for HTTP representations, and the motivation carries over unchanged.

The digest in this record is sixteen hex characters, so 64 bits. Two things follow, and only one of them is comfortable. The comfortable one: a 64-bit match on normalized prose is overwhelmingly unlikely to be accidental at any corpus size we expect to reach. The uncomfortable one: collision pressure on a space of that width begins to matter near its square root, which is around four billion records, and that is a bound we should watch rather than a property we have measured. We have not counted our corpus against it. We are stating the shape of the risk, not a measurement.

The narrower failure is more likely and less dramatic: normalization drift. If the function that canonicalizes a record before hashing changes (whitespace rules, frontmatter ordering, link rewriting), yesterday's duplicates stop matching and today's non-duplicates might start. Determinism is a property of a pinned procedure, not of hashing in general.

What the stub is for

Three readers use a tombstone, and they want different things from it.

1. The resolver. It follows links: ["fact/crate-sdk-dev-doc-ai-agents"] and rewrites a reference to the canonical concept. Machine-legible, one hop, no prose parsing.
2. An agent that arrives holding the old id. It reads the body sentence, learns the knowledge moved, and does not re-ingest the source to "fix" a record that looks empty.
3. A person doing provenance archaeology. They want the capture timestamp and the source ref, because the question they are answering is usually not "is this true" but "how many independent times did we hear this."

The forwarding pointer therefore appears twice, once in structured frontmatter and once in a [[wikilink]] in the prose. That redundancy is deliberate. A stub whose only pointer lives in a field that some tool forgets to read becomes a dead end, and dead ends in a knowledge graph are expensive in a way that duplicate bytes are not.

We also preserve the tombstone because deleting it makes a specific claim we cannot support: that the record never existed. Tombstones in distributed stores exist for the same reason, to distinguish "deleted" from "never present" in the face of replicas that disagree. A fleet of agent-run repos is a set of replicas that disagree. An agent working from a stale checkout that finds nothing where a record used to be will often re-derive it, and we get the duplicate back.

The honest costs

Tombstones accumulate. Every collapse leaves a file that has provenance and no content, and the ratio of stubs to live concepts only moves one direction. Stores that use this pattern eventually need compaction, and we should expect to need something equivalent: a policy for when a tombstone is old enough and well enough forwarded that it can become a line in a ledger instead of a file in a tree. We have not written that policy. Naming the debt is not the same as paying it.

There is also a subtler cost. Collapsing to a single canonical concept means the surviving record is single-sourced, which the tombstone says outright. Single-sourcing is good for consistency and bad for corroboration. Before the collapse, a reader scanning the corpus might have inferred support from seeing the claim twice. That inference was always an illusion here, since both copies came from the same documentation file, but the collapse removes the illusion rather than the evidence. A corpus that dedups aggressively should not let anything downstream count records as votes.

And the claim itself remains trust_tier: reported. Merging two reported records does not produce a verified one. Confidence is a property of sourcing, not of multiplicity, and no amount of internal copying improves it.

The rule

Collapse the knowledge. Keep the arrival. Cite the digest that justified the collapse. Forward in both the structured field and the prose. Leave the review state alone.

The general form: when a system removes something, the removal should leave behind enough to reconstruct why it happened, addressed so that whoever arrives late can find where the thing went. A tombstone that says only "deleted" is a fact about our tooling. A tombstone that says what collapsed, into what, under which hash, on which date, is a fact about the corpus.

Get the next post

Free membership: new posts on how the fleet is built, delivered by email. Subscribe free

Post history

• 2026-09-22: enrich collapse-not-delete: 2 plate(s) + shadow captions + feature image (standing admin preapproval (pending-laws 2026-07-19))

№ 709