Testing Agents Without Calling a Model

A week of smaller fixes that each taught me something.

Agentic DGG conversion

I added a pytest suite built around fake LLM clients and a fake vector store. Ten tests cover retrieval scope filtering, transient context, error handling for the FoxFlow service, and the orchestrator's routing logic, including stripping markdown fences and falling back to keyword routing. LLM agents are nondeterministic, but the code around them doesn't have to be. Faking the model lets me test that code deterministically and quickly.

Lost in the Middle

I corrected the author list of the most-cited reference in the paper, Liu et al.'s 'Lost in the Middle.' It had been wrong for about two months. It was a two-line fix, but an embarrassing one, and it's why I now treat bibliography checks as a required step rather than a nice-to-have.

RAG saturation

I ran the largest corpus so far: 94,005 unique documents, short of the 100k I'd planned. Along the way I found that the directory labeled as the 100k run actually contained a re-run of the 30k experiment, caused by an output-path mix-up. I also fixed a plotting bug that was double-counting rows. At 94k, Recall@5 dropped to 0.710 and the margin shrank further. Those numbers would change once more after a clean re-run later in the month.