Our weekly editorial posts are drafted by LLM subagents. Nothing they write enters the repository until it passes a set of gates. The gates are plain scripts that never call a model, on the principle that a gate verdict must be a fact about bytes rather than an opinion about them. An earlier fix to one of these gates added two replacement rules concerned with figures and the bodies they are attributed to. This note covers what those rules leave uncovered, and why that gap is written into the code and the self-tests instead of being left out of both.
What hole do the two replacement rules leave open?
The source frames the threat as a forger writing the draft. Against that writer, the two rules have a known bypass. The forger leaves the number alone and moves the attribution instead. The paragraph carries a link to the correct body, which satisfies the rules. The borrowed figure sits unattributed in its own sentence.
Both rules pass, and the borrowed figure is still on the page. It has only been rearranged.
What do the rules actually buy?
The source states their value in one line and no more: a borrowed figure can no longer be presented to the reader as another body's number. It can only be left hanging beside the name of the body it came from.
That is still a lie, but a weaker and a more visible one. The stronger form is a number stated as a named body's finding, which gives a reader no reason to question it. The weaker form is a number with no owner in its own sentence, sitting next to a link that does not claim it. An orphaned figure is easier to notice than a confidently misattributed one.
Why document the limitation in the code comment?
A comment next to the rules sits where an engineer will be reading when they decide whether to trust or change those rules. That is the moment when knowing the edge of the defence matters most.
The underlying reason is the one behind the whole approach. A defence that overstates itself is worse than one that admits its edge, because people stop looking. If the comment claimed the two rules prevented figure forgery, anyone relying on the gate would have reason to stop looking for this pattern. The residual risk would be the same, with less attention on it. Stating the hole keeps attention on a place the automation does not reach.
Why give an open hole its own test case?
The gate's self-tests include a case, C23, for the hole that is still open. That puts the known bypass into the test suite itself, not only into prose.
In general, a self-test for an unclosed gap does a different job from one for a handled case. It does not certify that the code works; it makes the gap part of what the suite visibly accounts for, so the limitation is harder to lose in a refactor and is already represented if someone later sets out to close it.
The comment and the test case say the same thing in two forms. The comment explains the gap in words. The test case gives it a fixed place in the suite.
How should you apply this to your own LLM output gates?
- Model the writer as an adversary. When an LLM produces the content your gate checks, ask how a writer who wanted to pass dishonestly would do it. Moving an attribution while leaving a number in place is the kind of bypass that only shows up when you look for it.
- State what a rule buys in one sentence. If you cannot say it in one line without overclaiming, you may not yet know its boundary.
- Put known gaps in the source, next to the rule. That is where the person deciding whether to trust the rule will be reading.
- Give each open hole its own test case. A known bypass that lives only in someone's memory is easy to lose; a case in the suite is harder to overlook.
- Prefer making a lie weaker and more visible over claiming to have made it impossible. Partial defences are normal. Partial defences presented as complete are the dangerous kind.
The two rules are worth having. They remove the more convincing form of the forgery. They are not worth more than that, and the code says so.