A Concept Channel That Scored Zero, and a Report That Said Otherwise

This week every project hit a release or results milestone, and one of them taught me to distrust my own report generator.

Concept-guided RAG

I ran the full harness on SciFact (210 test queries) as a smoke test, with sentence-transformer embeddings and a deterministic mock in place of the concept-extracting LLM. The baselines behaved sensibly: dense retrieval scored nDCG@10 of 0.627, multi-query and HyDE were a hair higher but not significantly, and BM25 was significantly worse at 0.563.

The concept channel scored essentially zero. So did the oracle and the shuffled control, which tells me the mock concepts carry no concept-specific signal at all; a simple keyword baseline scored 0.303. The validation split picked a fusion weight of 1.0, which means the fused retriever was just dense retrieval.

My report generator still printed a tentative positive outcome, and described the content and concept scores as weakly correlated when their correlation was -0.96. Both came from thresholds that didn't account for sign or context. The concept question is still unanswered, and the next step is real LLM concept extraction plus a fix to the interpretation logic.

Agentic DGG conversion

The emitter now outputs only the sections the current FoxFlow parser accepts (types, parameters and rules), with the rest behind an opt-in flag. With that change, all six golden cases parse against FoxFlow v0.3, edge syntax included. I emit less in order to emit something valid. The test suite is at 102 tests.

Lost in the Middle

I prepared the public-facing documentation: a scope statement saying plainly that the heuristic U-curve is there by design and that the real-model evidence is an eight-example pilot, plus a license, a citation file and a docs index. I also fixed a script that wrote every Ollama variant to the same directory, so each run silently overwrote the last. After the formatting fixes, I re-ran the four pilots and the aggregate scores matched July's exactly.

RAG saturation

I wrote a reviewer-risk memo of the paper's weaknesses: the entropy diagnostic is flat everywhere, there are only three corpus sizes with one run each, corpus size is confounded with composition, 'collapse' overstates a five-point recall drop, and my phrasing perturbations are close to null. All 69 tests pass and all 15 citations resolve.