Replacing an Invented Number with a Weaker Real One

The theme of this week was auditing my own claims.

RAG saturation

I wrote a manual review guide for the paper: for every claim, which file or run backs it. Going through it showed that the rank-stability figure I'd written in May (an average Jaccard of 0.68 under phrasing changes) had never come from an actual run. I removed it and ran the experiment instead: phrasing variants on the 30k SciQ corpus gave Jaccard@5 of 0.81.

There's an important caveat. My 'phrasing variants' are still mechanical edits to the query rather than real paraphrases, so 0.81 is a weak lower bound on instability, not a measurement of paraphrase sensitivity. A real number with honest framing beats an impressive one I can't defend.

I also expanded the manuscript, with a new title focused on discriminability collapse and rank instability, a hubness section in related work, an algorithm box and appendices. I redesigned the plots so toy and unique-corpus results are clearly separated.

Lost in the Middle

I revised the paper with an explicit section on its limitations: eight examples per run can't support significance testing, every per-bucket score is binary, and any citation of these results needs to say they come from a heuristic. I also wrote a checklist a reviewer can follow to verify each claim, and removed a reference that didn't hold up.

Agentic DGG conversion

I added an Ollama client with the same interface as the original LLM client, plus a provider switch. Being able to run everything locally means I can iterate without depending on an institutional API key.