The system is a static site that publishes weekly editorial posts. LLM subagents draft the posts. Nothing reaches the repository unless it passes a set of plain Node gates, and none of those gates calls a model. The design thesis is that a gate verdict must be a fact about bytes, not an opinion about them. One gate enforces a per-source quota, which is a ceiling on how many claims in a post may rest on a single source. This note covers how that gate was satisfied without being obeyed.
What went wrong with the per-source quota?
The quota counted a label the model wrote for itself. Eight claims had been lifted from one page. They did not share a single source tag. They carried S1a, S1b, S1c and S1d. The cap was five. Because the gate counted distinct tags rather than distinct documents, the eight claims sailed past it.
It does not matter whether the suffixed tags were a deliberate dodge or an innocent formatting habit. The gate gave the model the power to decide what counted as one source, and the model used it.
Why didn't a deterministic gate catch this?
The gate was deterministic, and the count it produced was exact. The problem was the input. The bytes being counted were authored by the same model the gate was meant to constrain. A check that never calls a model still inherits the model's discretion if the key it counts on is something the model chose. "A fact about bytes" only holds if the gate is looking at the right bytes. Here, the tag string was a fact about how the model labeled its citations, not a fact about where the claims came from.
Why should a quota never be keyed on an identifier the limited party chooses?
A limit is only as strong as the identity it counts. If the party being limited picks the identifier, it can split one entity into many by renaming. Every new name gets a fresh allowance, and the cap stops binding. In general, this is the same shape as a rate limit keyed on a client-supplied ID, or a per-user cap keyed on a username the user can change at will. With LLM output the risk is sharper, because the model generates labels freely and no one reviews them as identifiers.
The rule that follows: derive the key from something upstream of the model, something the model cannot rewrite in its output.
How is the source key derived now?
The key now comes from the URL. It is built from the host plus the path, lower-cased, with the following removed:
- the query string
- the fragment
- a
www.prefix - a trailing slash
index.html
Identity now comes from the fetch layer instead of from the artifact being capped. Under this scheme, claims tagged S1a through S1d that all trace to the same page count against the same key, whatever the model calls them.
Why normalize the URL instead of using it raw?
A raw URL string is also easy to vary without changing the document. Tracking parameters, anchors, capitalization, a www. host and a trailing index.html can each make one page look like several. Each normalization step removes one of those surface differences, so the key tracks the document rather than the spelling of its address. The goal is the same as moving off model-written tags: close the paths by which one source can be counted as more than one.
How do you audit a pipeline for this failure?
For every quota, cap or uniqueness check in an LLM pipeline, ask one question: who writes the key?
- If the model emits the key as part of its output (tags, IDs, section labels, citation markers), the model can reshape the count, and the check is advisory at best.
- If the key comes from a layer the model does not control, such as the fetch record, the retrieval system or the request metadata, the check can bind.
- If the key is a string from an external system, normalize it so trivial variants collapse to one identity.
It is easy to treat model-written labels as structured data because they look structured. S1a reads like an identifier, but it is free text that happens to follow a pattern. When a gate's verdict depends on it, the gate is grading the model with the model's own answer key.