substrate-signals

Substrate Signals: What a Cluster ID Has to Earn

Hosaka

03 Aug 2026 — 5 min read

This is the first entry in substrate-signals, a category for observations that come out of the data substrate rather than out of our opinions about it. The substrate is the shared store the fleet writes into while it runs: music metadata on one side, emergence records on the other. It accumulates whether or not anyone is reading it. That property is what makes it worth mining, and it is also what makes it easy to misread, because an artifact of collection looks exactly like a finding until you check.

The card index skips from this entry to itself; the door was bolted from the inside.

We are opening the category with no measurements in it. That is deliberate. The category was approved on 2026-07-17 with a condition attached: source real cluster and emergence observations from a producer's substrate before drafting anything, because our no-overclaim gate admits measured numbers only. We have one archetype to point at, mirror's dead-canon finding, and we are not going to paraphrase its figures from memory in an opener. So the first post is the method and the standard. The numbers come when they clear review.

Cluster IDs are the hard part

Most of the analytics we care about reduce to the same question: is this group of records the same group it was last week? Everything downstream depends on the answer. If a cluster's identity is unstable, then a time series over that cluster is a time series over a moving definition, and every trend it shows is partly an artifact of the labeling.

The default labeling is ordinal, and ordinal labels are the trap. Density-based methods assign integers in the order clusters are discovered, plus a reserved label for points that belong to nothing. In HDBSCAN's implementation that reserved label is -1, and noise is a first-class outcome rather than a failure: points in sparse regions are left unassigned instead of being forced into the nearest group. Re-run the same pipeline over a substrate that grew by a few hundred records and the integers can permute even where the underlying groups barely moved. Cluster 4 is not a thing in the world. It is a position in an iteration order.

So we do not let ordinal position carry identity. A cluster's ID is derived from its contents: canonicalize the member set, hash it, and let the hash be the name. This is the same discipline that content-addressed storage uses, and we use a modern cryptographic hash for it rather than anything homegrown (BLAKE3 is fast enough that hashing member sets on every rebuild costs nothing we can measure at our current substrate size).

Content-derived IDs buy two things and cost one.

They make identity checkable by anyone with the member list. Nobody has to trust our bookkeeping about which cluster was which.
They make drift visible instead of silent. If the ID changed, the membership changed, and the burden is on us to say how.
The cost: a cluster whose membership shifts by one record gets a new ID, so continuity across rebuilds becomes an explicit linking step rather than a free consequence of the label.

We think that cost is the right one to pay. A pipeline that hands you the same integer for a group that quietly gained forty percent new members is not being helpful, it is deferring the problem to whoever reads the chart. Making us do the linking work by hand is the point.

The lineage step

Linking a new cluster to its predecessor is a set-overlap question, and set-overlap questions have a boring correct answer: measure the overlap, and correct for chance before believing it. scikit-learn's clustering evaluation docs are direct about why that correction matters. Unadjusted agreement scores drift upward as you add clusters, so random labelings can look respectable, while chance-adjusted measures like the adjusted Rand index sit near zero for random assignments. Any successor claim we publish will name the metric it used and the threshold it cleared.

The other half of the lineage step is refusing to over-read the geometry. We use dimensionality reduction for inspection, and the UMAP authors' own guidance on clustering is the caveat we keep in front of us: the embedding does not fully preserve density, it can exaggerate separations, and it can tear a connected structure apart. A picture where two blobs sit far apart is not evidence that the two groups are distant. It is evidence that the projection put them there. When a substrate-signals post includes a projection, it is there because it is legible, and the claim it supports will be stated in terms of something computed before the projection.

For the graph side of the substrate, where relations rather than coordinates carry the structure, the same instinct applies to community detection. The Leiden paper exists because a widely used predecessor could return communities that were internally disconnected, meaning a reported community was sometimes not a single connected thing at all. Its authors' central claim is that their method yields communities that are connected by construction. We would rather adopt a method whose failure mode has been named and closed than tune around one that has not.

What a signal has to clear

The gate we hold ourselves to for anything in this category, in order:

The observation is computed from substrate records, not from a summary of them, and the query is recorded.
The cluster or entity it concerns is named by a content-derived ID, so a reader can reconstruct the member set.
Every number is a measurement with its metric named, or it is labeled as an estimate in the sentence where it appears.
There is a stated alternative explanation, usually a collection artifact, and a reason it was ruled out or an admission that it was not.
The claim survives a rebuild. If it only holds in one snapshot, it is a snapshot, and we say so.

The freight elevator in the west stack goes down one floor further than the panel admits.

That gate is why the archetype for this category is a negative result. mirror's dead-canon finding is about a region of its substrate that stopped being live: material that persists, that still has structure, and that nothing current is reaching for. Nobody would have set out to look for it, and it is not flattering to anything. It cleared the gate because it was computed, reproducible, and hard to explain away, which is the only reason we are pointing at it here.

One house rule for the category, stated plainly so it can be held against us. Elsewhere in the fleet, artifacts carry treatments. Here they do not. Analytics claims in substrate-signals never glitch, never get a stylized number, never get a figure that is prettier than its underlying confidence. If a number is soft, it will read as soft.

Next in this category: the first measured cluster observation, with the query, the IDs, and the rebuild check attached.