When the retriever changes, do the answers get better?
A question-answering system over recent arXiv papers, built eval-first: a frozen golden set and its scorers came before the pipeline, and every retrieval rung was measured with the same harness, prompt, top-k and models. This page reads the committed results; nothing here calls a model.
- 150 CC BY 4.0 papers submitted after the generator's knowledge cutoff
- 140 questions, hash committed before the first run
- Run-to-run noise measured on three identical runs
The retrieval ladder
Grounded answer accuracy (GAA): an answerable question answered correctly and faithfully, or an unanswerable one declined, over all 140 questions. The shaded band is BM25's spread over three identical runs (0.014); a gap inside it is noise.
Loading the committed results…
Question explorer
Pick a question to see the gold answer, its evidence quote, and what each rung answered with the judge's verdict. The squares show pass or fail per rung (none, BM25, dense, hybrid, rerank).
Loading…
What these numbers do and do not say
- The judge is partly calibrated. Against one person on 30 items, its correctness verdicts agreed on 29 (kappa 0.91) and its faithfulness verdicts on 27 (kappa 0.53, below the 0.6 bar), so GAA, which needs both, is not fully calibrated.
- The ranking held under other judges. Re-judged by claude-sonnet-5-5 and claude-haiku-4-5, the order BM25 > hybrid > dense > rerank stayed the same.
- The golden set was drafted by a model and kept only when its quotes were found verbatim and a blind re-answer agreed; it has not been checked question by question by a person.
- Three runs give a rough noise floor, not a confidence interval. The rerank repeat moved GAA by 0.000 but flipped 6 of 140 individual verdicts.