Silent Defaults Are the Real Threat to Validity

I spent this week hardening all three projects, and the most important bugs were defaults I never chose on purpose.

RAG saturation

I found that baseline mode quietly forced 200 queries, sampled with replacement from 100 unique SciQ questions. My published results used that setting, so 'n = 200' really means 100 unique questions. The default now respects the configured subset size and warns when it has to oversample.

I also fixed a data-prep fallback that inserted MS MARCO queries as documents, made the pipeline fail fast when relevance IDs don't join the corpus, recorded dependency versions in every run's fingerprint, and added CI. The 94,005-document count now has a proper explanation: 11,679 SciQ passages plus 82,326 MS MARCO passages.

Lost in the Middle

CI now runs the tests on Python 3.10 through 3.12. The Ollama adapter used to swallow API errors and return an empty answer, which would have been scored as wrong. It now raises. None of my committed predictions were empty, so the results weren't affected, but the bug could have quietly lowered future scores.

I also found that the YAML template for passage formatting inserted blank lines between passages, so my July runs used slightly different prompt formatting than intended. Every run snapshot now records the git commit and Python version.

Agentic DGG conversion

The transpiler's response time swung from 33 seconds one day to 292 seconds the next, so I raised the timeout to five minutes and added another failure message format to the detector. Waiting five minutes before falling back is a real cost, but stopping early meant giving up on responses that would have succeeded.