30k Documents and a Shrinking Margin

After a quieter week, the RAG study got its second data point.

RAG saturation

The 30k-document run (SciQ mixed with MS MARCO) gave Recall@5 of 0.735, down from 0.745 at 11.7k, and a slightly smaller score gap. A small move, but in the direction the hypothesis predicts.

I also did a round of repository hygiene: a README rewrite, an MIT license, command-line arguments for the data preparation scripts, and unit tests for retrieval metrics and stability. Along the way I changed how the score gap is computed, from top-1 minus the k-th score to top-1 minus the mean of the top k. I didn't re-run the earlier numbers at the time, which set up a reproducibility puzzle I only untangled in July.

Lost in the Middle

The reproduction report now has a proper Limitations section. Its three main points: the heuristic hard-codes the effect, the synthetic data is simple, and I measure position by passage index rather than token distance. Writing those down plainly made me notice that my conclusion still said I had 'confirmed the effect.' The limitations and the conclusion were telling different stories, and the limitations were right.

Reflection

Changing a metric definition without re-running old results is a quiet way to make your own tables inconsistent. I now treat metric definitions like an API: version them, and regenerate everything downstream when they change.