Auditing My Own Table

I tried to reproduce my own main results table from scratch this week, and it didn't fully reproduce.

RAG saturation

The audit turned up three problems. First, the 11.7k row of the table had been hand-assembled: the recall number came from one run, and the score margin came from a different run where document IDs didn't line up. Second, re-running the 30k configuration reproduced recall, similarity and entropy exactly, but the score margin came out at 0.087 instead of 0.114, consistently across several runs. Third, some outputs I needed weren't available locally.

I deliberately left the table untouched until I could re-run everything cleanly in a pinned environment, and pinned exact dependency versions in the meantime.

Lost in the Middle

I added an Ollama adapter so I can finally run a real model locally, with no API key. A smoke test passed: the model answered 'Jupiter.' and the gold answer was 'Jupiter.'

That smoke test highlighted a risk worth writing down before seeing any results. Strict exact-match scoring can mark a correct but wordy answer as wrong. I recorded the rule I'll follow: report what actually happens, don't hand-clean individual predictions, and call out metric artifacts explicitly.

Reflection

Deciding in advance how to handle inconvenient results is the closest thing to pre-registration I can do as an independent researcher, and it already feels like it's keeping me honest.