AutoIndex: Learning Representation Programs for Retrieval

Abstract

We present AutoIndex, a framework for learning representation programs: executable transformations that map raw documents into the representations exposed to a retrieval system. Rather than tuning retrievers, rerankers, or a small set of preprocessing hyperparameters, AutoIndex searches over programs that slice, enrich, normalize, reweight, or reorganize documents before indexing. At each iteration, agents diagnose failures of the current program and synthesize candidate updates, retaining only updates that improve retrieval quality under the resulting index. We evaluate on CRUMB, a benchmark of heterogeneous retrieval tasks, with BM25 held fixed across all experiments. The learned programs improve recall on all 8 tasks, with average gains of +8.4% Recall@100 and +8.3% nDCG@10, and largest gains of +30.5% and +43.6%. These results suggest that document representation should not be treated as a fixed preprocessing choice made before retrieval begins, but as an explicit optimization target.

Section 1 · Method

Program search as a closed optimization loop

Each iteration diagnoses where retrieval is failing, synthesizes candidate document-representation programs, rebuilds the index, and keeps only what improves validation Recall@100.

01 · DIAGNOSE

Analysis Agent

Inspects why queries fail and writes a grounded summary.

02 · SYNTHESIZE

Code Agent

Proposes executable programs that transform documents, stripping noise, reweighting sections, re-chunking.

03 · SELECT

Verify & keep

Only real improvements on held-out validation survive.

The retriever, ranking rule, and indexing backend are all held fixed. Every gain comes from the learned representation program, applied at indexing time with no further LLM calls.

Section 2 · Results

Better retrieval on every task

We evaluate on CRUMB, eight complex retrieval tasks with long documents and heterogeneous evidence, holding BM25 fixed throughout. Bars show relative Recall@100 improvement over a full-document BM25 baseline, for both code-generation backbones. Hover any bar for its exact gain and underlying scores.

Claude Sonnet 4.6 (n=2) qwen3-coder (n=3)

Δ Recall@100 vs. the BM25 full-document baseline across CRUMB splits. AVG is the unweighted macro-average.

Line chart of Δ nDCG@10 vs. BM25 baseline across search iterations 0 to 4, for StackExchange, SetOpEntity, and ClinicalTrial. Filled points mark adopted candidates, open points mark rejected candidates. StackExchange and SetOpEntity climb steadily as the search adopts improving programs; ClinicalTrial plateaus early. — **Gains accrue over the search, not in one step.**

AutoIndex transfers to dense retrieval as well. A single dense-retrieval probe on StackExchange with Qwen3-Embedding-0.6B reused the learned representation and lifted held-out Recall@100 from 0.739 → 0.874 (+18.3%), an early sign that learned representations aren't tied to the retriever that produced them.

Cite

BibTeX

@misc{onuallain2026autoindex,
  title         = {{AutoIndex}: Learning Representation Programs for Retrieval},
  author        = {Sam O'Nuallain and Nithya Rajkumar and Ramya Narayanasamy and
                   Hanna Jiang and Shreyas Chaudhari and Andrew Drozdov},
  year          = {2026},
  eprint        = {2607.18603},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR},
  url           = {https://arxiv.org/abs/2607.18603}
}