ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
Abstract Overview
ScholarCatalyst is a benchmark for retrieving prior papers whose ideas did or could have meaningfully advanced a research project. The benchmark is built from firsthand judgments by 184 lead authors covering 207 recent computer science papers, using an automated pipeline that prepares candidate papers and queries for author validation. It evaluates whether systems can retrieve these catalyst papers from a 191K-paper corpus restricted to literature available before each project was completed, using early-stage research questions rather than solution-aware queries. Across lexical, dense, multi-vector, and agentic methods, current systems recover only a limited fraction of author-identified inspiration papers.
Novelty
The main novelty is a retrieval benchmark grounded in authors' firsthand accounts of which prior papers actually inspired or could have advanced their work, including rationales for those judgments. The task uniquely combines pre-discovery queries, author annotations, and latent relevance signals that are largely absent from formal citations and the published record, supported by an automated pipeline that keeps the benchmark refreshable.
Results
General-purpose dense retrievers achieve the best performance among evaluated pre-cutoff models, yet reach only 39% Recall@20 on core queries and 51% on subfield-specific queries. Agentic search methods fail to outperform standalone embedding retrieval (0.42 vs. 0.48 Recall@20), primarily due to constrained candidate coverage during multi-turn searches. Furthermore, author-identified catalyst papers exhibit no greater lexical or semantic similarity to the queries than rejected hard negatives, demonstrating that topical similarity is insufficient for this task.
Key Points
- ScholarCatalyst contains 894 queries derived from 207 projects, paired with author-validated positive and hard-negative papers drawn from a 191K pre-project computer science corpus.
- Current retrieval and agentic systems struggle on catalyst retrieval, with the strongest pre-cutoff dense retrievers missing roughly half or more of gold papers in their top-20 results.
- Analysis reveals that scientific inspiration relations are diverse, frequently uncited, and weakly reflected in text similarity metrics, making candidate generation a primary bottleneck.