Pipelines That Work but Measure Nothing (Yet)

This week was about stress-testing the harnesses I built in the first two weeks, and the main lesson was humbling: a pipeline can run end to end, produce clean tables, and still not be measuring anything yet.

RAG saturation

I wired a QueryPhrasingShiftGenerator into the pipeline and added a rank-stability metric: Jaccard overlap of the top-k results across phrasing variants. Then I ran my first scale sweep at 1k, 10k and 100k documents.

Every scale came back identical: Exact Match 1.0, Jaccard 1.0, a mean top-k similarity of 0.6204, and a top-1 score gap of exactly 0.0. At first glance that looks like a retriever that never degrades. In reality my 'scaled' corpora are about twenty base documents duplicated over and over, and the generator is a mock that extracts a planted answer. Of course nothing moves.

Lost in the Middle

On the reproduction track I added a tuned configuration that widens how much of the beginning of the context my heuristic model reads. Overall EM went from 0.50 to 0.75, and the early-middle bucket jumped from 0.0 to 1.0.

I want to be precise about what last week's 'U-shape' result means. The heuristic model only reads the first and last passages by design, so the curve is built into the model rather than discovered by the experiment. What it validates is the pipeline: position bucketing, prompt construction, and per-bucket metrics all behave the way they should. The real question, whether a neural model shows the effect without being told to, is still ahead of me.

Reflection

Both projects taught me the same thing this week: a null result from a degenerate setup isn't a finding, it's a bug report about the experiment design. Writing down exactly why the numbers look the way they do is turning out to be as valuable as the numbers themselves.