Adding Noise to a Signal That Wasn't There

Third week in a row of null results on the RAG side, and that was the push I needed to change direction.

RAG saturation

I built a NoiseInjectionShiftGenerator that mixes filler documents into the corpus at factors from 0x to 2x. Every factor produced the same metrics: EM 1.0, Recall 1.0, Jaccard 1.0, and a gap of 0.0. Because the corpus is mostly duplicates of the relevant documents, injected noise never even reaches the top 5.

Three nulls in a row made the diagnosis clear: the corpus design itself is the bottleneck. To measure saturation I need a corpus of genuinely unique documents, so next up is a loader for external datasets.

Lost in the Middle

I added a distractor-count override and ran a 'noise shift' with 10 distractors (11 passages in total). The pattern held: 0.50 overall, perfect at the edges, zero in the middle, with the middle buckets now landing at different indices.

A subtlety I didn't appreciate at the time: my examples only have four unique distractors, so filling ten slots means repeating them. The context gets longer, but not as much more varied as it looks. I later had to rewrite my claims about context length because of this.

Reflection

Experiment design has become the hardest part of this research. Writing the code is fast; making sure the experiment can actually distinguish the hypotheses is slow.