The project behind this note is an Android app that watches a child's chat apps on the phone itself. It sends only suspicious slices of conversation to an LLM that decides whether an adult is grooming the child. When the LLM decides that, the app tells the parent or locks that one chat. Its evaluation methodology contains a statistical rule that applies to any agent eval: an ablation comparing memory against no memory is a paired design, and the choice of test follows from that.

What makes a memory-vs-no-memory ablation a paired design?

The methodology classifies the ablation as paired. It requires McNemar's exact test or Wilcoxon signed-rank, and it rules out two-sample tests. The claim does not describe how trajectories are run under each condition, so this note does not describe that procedure either.

The following is textbook background, not a description of this project's procedure. A design is called paired when each unit, here a trajectory, contributes an outcome under both conditions, which links the two outcomes. A two-sample test treats the arms as independent groups and discards that link. A paired test works on the difference within each unit.

Which paired test applies: McNemar's exact test or Wilcoxon signed-rank?

The methodology names both. As standard background, McNemar's exact test is the usual choice for paired binary outcomes. Pairs where both conditions agree drop out, and only discordant pairs enter the test:

Wilcoxon signed-rank is the standard paired test when each unit yields a numeric difference between conditions. The claim does not say which outcome measure each test covers in this system, and this note does not guess.

Why can a 40-60 trajectory ablation be meaningful when a 60-trajectory recall claim is not?

The methodology states that a paired ablation of 40-60 trajectories can be meaningful, while a 60-trajectory absolute recall claim is not. It ties that difference to the paired design. The sample sizes overlap, so the deciding factor is not the trajectory count.

What follows is general statistical background. It offers one way to read the statement and is not the methodology's own argument. In general, pairing removes variation between units. A trajectory that is easy or hard tends to be easy or hard under both conditions, so a paired test mostly sees the disagreements. An absolute recall estimate has no second condition to difference against, so variation between trajectories stays in the estimate. That is a common reason the same order of sample size can support a difference claim but not a level claim.

What does (b,c)=(1,0) at N=2 tell you?

The current ablation has N=2, with b=1 and c=0. That means one discordant pair in one direction and none in the other. The project publishes these counts itself and calls the result a demo, not evidence.

The label matches the numbers. McNemar's exact test uses only discordant pairs, so the whole comparison rests on a single trajectory. One flip cannot separate an effect of memory from chance. N=2 is also far smaller than the ablation sizes the methodology describes as able to be meaningful.

How should a team report a small paired ablation?

These practices follow from the reasoning above. They are general recommendations, not a description of this project's reporting process.

Before you size an eval set, decide which kind of claim you are making: a difference between conditions or an absolute level. The same number of trajectories can be enough for the first and not enough for the second.