First Successful Handoff, and a Retrieval Bug Hiding in PDF Glyphs

A big week: a first success on the DGG project and the start of a new one.

Agentic DGG conversion

On August 31 the agent path produced its first successful FoxFlow conversion through the live transpiler. That success also showed that my failure detector was wrong. I had concluded the week before that a zero token count reliably meant failure, but the service reports zero tokens on success too, which made the agent path unreachable. I now classify responses by their content, and later by a structured success field the service added.

Then a request for 'a particle loop in a circle' came back as a microtubule model. I traced it to two causes. PDF extraction had filled the index with junk: hex escapes in more than half the chunks, broken ligatures, and glyph names from figures. One of those glyph names was literally 'CIRCLE', so the word matched microtubule figures. On top of that, my 32-character task was buried behind about 2,600 characters of retrieved context. I added text repair at ingestion, which shrank the corpus from 1,323 to 908 clean chunks, and restructured the prompt to state the task first, cap the context, and restate the task at the end.

I also installed Julia and ran FoxFlow's public parser. It rejected all six of my files, along with an example that the deployed service accepts, at the same point: named attributes. The public repository is behind the deployed transpiler, so it can't be the final judge of validity yet. The test suite now has 95 tests.

Concept-guided RAG

I started a fourth project. The question: do LLM-extracted concepts add a ranking signal beyond content similarity, especially when a query and its relevant document share few words? Documents are scored as a weighted mix of content and concept similarity, with the weight chosen only on a validation split.

Before touching real data I built the whole evaluation harness: BM25, dense, multi-query, HyDE, concept-only and fused retrievers, an oracle-concept upper bound, paired significance tests with multiple-comparison correction, and a log of design decisions. By the end of the week it also had a shuffled-concept negative control and a keyword-based concept baseline. Building the controls first, before I knew what the results would be, was a direct lesson from the other two projects.

RAG saturation and Lost in the Middle

Both papers got tighter claims. The RAG paper now says its diagnostics anticipate drops in Recall@k, not in end-to-end answer quality, which I never measured on unique corpora, and it gained an explicit list of what it does not claim. The reproduction paper's own self-assessment now reads 'not yet publish-ready', and an earlier draft is clearly marked as superseded.