First Real-Model Results: The Effect Mostly Didn't Show Up

This was the week I'd been building toward since March, and the result surprised me.

Lost in the Middle

I ran Llama 3.2 3B locally through Ollama on all four experiment variants. Each run has eight examples, two per position bucket. Exact match came out at 0.875 for the baseline, 1.0 with redundancy, 0.875 with ten distractors, and 1.0 with topically similar distractors.

Only two of the 32 predictions were scored wrong, and both were on the same arithmetic question. In the baseline run the model answered '3. The answer is: 3.', which is correct but fails strict exact match. In the noise run it answered '5' to 'What is 5-2?', the only genuine error in the whole pilot. The late-middle position, where I expected the most trouble, was perfect in every run.

So with this model, these tasks and this sample size, the U-shaped curve mostly didn't appear. That's a real finding about the limits of my setup: the tasks are probably too easy, and eight examples is a pilot, not an evaluation. It's not evidence that position doesn't matter. It's the clearest argument yet for scaling up the dataset and testing harder tasks and other models.

RAG saturation

I reinstalled the pinned environment, regenerated the corpora, and ran all configurations back to back. The clean numbers: Recall@5 goes 0.745, 0.735, 0.695 at 11.7k, 30k and 94k documents, and the top-1 score margin shrinks from 0.093 to 0.087 to 0.079. The margin contraction is about 15%, not the 34% I had reported earlier from mismatched runs. The paper now includes a footnote explaining the correction.

Reflection

Neither result is as dramatic as I hoped, and both are more trustworthy than anything I had in May.