Building the Bridge to Real Models

This week I finally started building the piece the reproduction study has been missing: support for real language models.

Lost in the Middle

I added model adapters for an OpenAI chat model and a Hugging Face Llama 3.2 1B Instruct checkpoint, alongside the existing heuristic. I also built a harder dataset where the distractors share the question's topic, so the model has to find the right passage among plausible-looking ones rather than obviously irrelevant filler.

I wrote a sweep script that varies the number of distractors from 4 to 19 to study context length properly, and made per-bucket metrics compute automatically for every run.

None of the new adapters have produced results yet. The README says so explicitly, and I'd rather keep an honest 'pending' than fill the space with expectations.

Reflection

Infrastructure for the real experiment turned out to be the easy part. The harder part, which I underestimated, was getting myself to actually run it instead of polishing everything around it.