The First Non-Degenerate Number

This week the RAG project finally touched real data.

RAG saturation

I added a Hugging Face dataset loader and pointed the pipeline at SciQ, a corpus of unique science passages. For the first time the score distribution looked alive: mean top-k similarity of 0.418, a score gap of 0.069, and entropy just below its maximum.

Retrieval quality, though, was zero. EM and Recall were both 0.0 because I was still asking my toy questions, whose answers don't exist in SciQ. So the first meaningful number came from a deliberately mismatched setup. It still told me something useful: the retriever produces a real, non-degenerate ranking on unique text, and the next step is building questions that are actually aligned with the corpus.

I also wrote my first analysis and plotting script, which made the contrast between duplicated and unique corpora visible at a glance.

Lost in the Middle

I consolidated the baseline, redundancy and noise-shift runs into the experiment log with per-bucket tables. I marked those experiments 'completed' in the README. In hindsight, 'completed' meant the runs finished, not that the scientific question was answered, and I've become more careful with that word since.

Reflection

Alignment between queries and corpus is a precondition for any recall number to mean anything. It sounds obvious written down, but it's easy to miss when the pipeline happily returns results either way.