We keep an unusual artifact in our archive: a Python trading system written entirely by an AI from a single 240-line specification document. Both the spec and the generated code survive, so a requirement can be traced from its heading to the code it became. This note follows one heading, the leverage cap, and what came out of it.
What did the spec's leverage-cap heading turn into?
It became two separate implementations of the same rule. The first is an inline min() inside the position sizer. The second is a standalone LeverageCaps class that implements the rule as its own unit.
We are not assessing whether either implementation is correct, or whether the two agree on limits and edge cases. The finding is narrower: one rule, two implementations, and a split between what runs and what is tested.
Which implementation does production actually use?
Production uses the inline min() in the position sizer. No production code imports the LeverageCaps class. It exists in the codebase and is named after the rule, but nothing in the production path depends on it.
Why does it matter that the tests cover the dead implementation?
Here is how the two implementations compare:
| Implementation | Production callers | Unit tests |
|---|---|---|
Inline min() in the position sizer | Used by production | 0 |
Standalone LeverageCaps class | 0 | 2 |
The unit tests written directly against the leverage rule target the class that production never imports. Two consequences follow from that structure:
- The direct tests do not reach the live cap. Those unit tests do not call the inline
min(), so they say nothing about it. This count covers direct tests only. It does not rule out other tests, such as broader position-sizer tests, exercising themin()indirectly. What it does mean is that a reader looking for the leverage-rule tests will find tests for the wrong code. - Changes to the dead class can look like real changes. Someone could edit
LeverageCaps, watch its tests pass, and believe they had changed how production enforces leverage. With no production callers, they would not have.
Our opinion: for a risk control, tests that appear to target the rule but sit on unused code deserve more scrutiny than they usually get, because their presence can make the rule look covered during review.
How does a single requirement end up implemented twice?
We are not claiming a cause for this system. The record shows the outcome, not the reasoning that produced it.
In general, though, a spec heading names a rule without saying where that rule should live. Code generated while building a position sizer has a local reason to enforce a cap where size is computed. Code generated by working through a spec heading by heading has a different local reason: to produce a dedicated component with its own tests. Nothing in a behavior-only spec forces those two paths to be reconciled. When a spec describes behavior but not ownership, it leaves room for duplicates.
What should you check when reviewing AI-generated code built from a spec?
- Find every implementation of each requirement, not just the first. Search by behavior, meaning where the rule's limits are actually applied, not only by name. A class named after the rule is the easiest copy to find, and here it was the copy production never imports.
- Confirm production callers for each implementation. A module whose only importers are tests is either dead code or unfinished wiring. Decide which, explicitly.
- Map tests to the call path production uses. Tests that land on unused code add confidence without adding protection. For safety-critical rules, check that at least one test exercises the exact function the live system calls.
- Write ownership into the spec. A sentence such as "the leverage cap is enforced in one place, and that place is the position sizer" closes the gap that a bare heading leaves open.
The lesson from this artifact is simple. A spec heading does not guarantee a single implementation, and a passing test is only evidence about the code that test actually calls.