Six Months In: What Independent Research Has Taught Me

The retrieval saturation paper got its release surface this week: a citation file, a data availability statement, and a scope section in the README stating that every result uses a deterministic reader and makes no claim about end-to-end RAG quality. I also cleaned the internal working notes out of the public tree.

It's a good moment to look back on six months across four projects.

What the evidence actually says

On retrieval saturation: as unique corpora grow from 11.7k to 94k documents, Recall@5 falls from 0.745 to 0.695 and the top-1 score margin shrinks by about 15%. Redundant toy corpora hide this completely, looking perfectly stable while their score distributions have collapsed. On Lost in the Middle, I have a clean, tested harness and a pilot where a small model mostly didn't show the effect. On concept-guided RAG, the harness is built and the real experiment is queued. On DGG conversion, all six golden models produce FoxFlow that parses, and the agents have completed a real conversion, though not reliably yet.

What I learned

Design the experiment so it can fail. My biggest early 'results' couldn't have come out any other way: duplicated corpora, a heuristic that ignored the middle by construction, and a redundancy fix that couldn't miss.

Write claims after the data, not before. Several numbers I drafted in May never survived contact with the actual runs. The most valuable documents I wrote this year were the audits that walked them back.

Treat metrics, defaults and outputs like production code. A changed formula, a resampling default and an output-path mix-up each moved published numbers more than any modeling choice did.

Use AI tools, and verify everything they produce. Coding and writing assistants made me much faster, and they also confidently produced wrong citations and over-claimed results. Verification is now part of my process, not an afterthought.

What's next

Run real LLM concept extraction on SciFact, evaluate neural models on the full 200-example position dataset, decide on venues for both papers, and keep pushing the DGG agents toward reliable round-trip conversion.