Aligned Questions and a Real Recall Number

A milestone week for the RAG study: my first real retrieval-quality number.

RAG saturation

I wrote a preparation script that turns the first 100 SciQ questions into an evaluation set linked to their supporting passages. A second script builds larger corpora by mixing SciQ with MS MARCO passages.

On the 11.7k-document SciQ corpus, Recall@5 came out at 0.745 with a score gap of 0.122. That's the baseline every later scale comparison builds on.

I also started the manuscript, covering the abstract, introduction, experimental setup and distribution shifts. Writing forced me to articulate the core claim: as a corpus grows, the retriever doesn't fail loudly; the margin between the right document and its neighbors quietly shrinks.

Lost in the Middle

I drafted an extension analysis connecting the results to RAG design. My first draft said the effect 'scales linearly with context length' and recommended duplicating evidence in RAG prompts. Two binary data points from a heuristic model can't support a scaling law, and I later retracted that line. It's a good example of how quickly a tidy result turns into an overstated sentence.

Reflection

Starting to write while experiments are still running is useful because it shows the gaps in the argument. It's also risky, because prose has a way of running ahead of the data. More on that next week.