MUSES: A Benchmark for Prospective Intellectual-Roots Retrieval

• URL: http://arxiv.org/abs/2609.00313v1
• Date: Mon, 31 Aug 2026 20:05:36 GMT
• Title: MUSES: A Benchmark for Prospective Intellectual-Roots Retrieval
• Abstract summary: textbfMUSES is a million-instance benchmark for prospective intellectual-roots retrieval over a fixed 2.33M-paper corpus.<n>textbfCiteRoots pairs a rhetorical layer over local citation text with a paper-level author-endorsed layer.<n> MUSES organizes difficulty along two axes: a emphfamiliarity axis spanning CiteNext, CiteNew, and CiteNew-Isolated, and a emphfunctional axis spanning broad citations, rhetorical roots, and author-endorsed roots.
• Score: 6.572091819435388
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific discovery depends on finding prior literature that shapes what comes next. Existing retrieval systems optimize for relevance and popularity, often favoring central papers over less familiar works that later prove generative. We introduce \textbf{MUSES}, a million-instance benchmark for prospective intellectual-roots retrieval over a fixed 2.33M-paper corpus, with roughly 140K test instances per familiarity tier. To our knowledge, it is the first prospective benchmark at this scale with a shared retrieval task and author-confirmed paper-level root labels. Alongside it, \textbf{CiteRoots} pairs a scalable rhetorical layer over local citation text (LLM judge $κ= 0.896$ versus human gold) with a paper-level author-endorsed layer ($n = 1{,}518$ generative-inspiration pairs from 753 focal papers). MUSES organizes difficulty along two axes: a \emph{familiarity} axis spanning CiteNext, CiteNew, and CiteNew-Isolated, and a \emph{functional} axis spanning broad citations, rhetorical roots, and author-endorsed roots. Across 9 method classes, a lean multi-centroid retriever built on SPECTER2 is strongest. Hit@100 falls from 0.534 on CiteNext to 0.424 on CiteNew, 0.205 on rhetorical CiteNew, and 0.171 on author-endorsed CiteNew, a $3.1\times$ decline. In a registered eight-lens full-test audit, roughly half of broad-tier test instances remain unsolved at K=1{,}000. Rhetorical role and author endorsement are distinct: the same judge agrees with endorsement at $κ= 0.037$. We release MUSES, both CiteRoots layers, and a distilled open companion judge for future work on prospective retrieval and intellectual roots.

Related papers

• An Interactive Agent for Requirement-Driven Candidate Sourcing [75.63902836826038]
  We present sys, the first requirements-driven candidate-sourcing agent.<n>It elicits, validates, retrieves, and verifies a vague people-request into a justified slate.<n>It dominates breadth ($100%$ coverage at $2.5times$ the yield) and is emphnear-orthogonal to the field.
  arXiv  Detail & Related papers  (2026-08-24T17:04:16Z)
• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• Structure-Aware Semantic Chunking with Title-Chain Prefixes: A 1600-Query Evaluation and the Measurement Trap in Text-Transform Ablations [4.342406076796542]
  We present a three-stage, chunk-side-only semantic chunking pipeline.<n>The title chain reuses the document's own header hierarchy instead of a generated summary.<n>The pipeline improves MRR@5 from 0.374 to 0.463 on the full set and from 0.828 to 0.925 on the answerable subset.
  arXiv  Detail & Related papers  (2026-08-01T19:01:46Z)
• Catalyst Papers in Artificial Intelligence Research: A Landscape on ICLR from 2017 to 2025 [5.523132953818281]
  OpenReview now makes numeric reviewer scores and accept/reject decisions public for every ICLR submission.<n>We compare four disruptiveness measures (the Consolidation/Destabilization (CD) index, node2vec, the direction-aware Embedding Disruptiveness Measure (EDM)) and define a five-type operational catalyst taxonomy.
  arXiv  Detail & Related papers  (2026-05-24T05:52:10Z)
• MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature [14.082854604879572]
  MasterSet is a large-scale benchmark designed to evaluate must-cite recommendation in the AI/ML domain.<n>It incorporates over 150,000 papers collected from official conference proceedings/websites of 15 leading venues.<n>The benchmark task requires retrieving must-cite papers from the candidate pool given only a query paper's title and abstract.
  arXiv  Detail & Related papers  (2026-04-20T00:34:28Z)
• Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs [75.62489398666644]
  We show that an emphautoresearch-style pipeline citepkarpathy2026autoresearch powered by Claude Code discovers novel white-box adversarial attack.<n>textbfsignificantly outperform all existing (30+ methods) in jailbreaking and prompt injection evaluations.
  arXiv  Detail & Related papers  (2026-03-25T16:50:56Z)
• ClaimFlow: Tracing the Evolution of Scientific Claims in NLP [67.23189226608389]
  We introduce $texttClaimFlow$, a claim-centric view of the NLP literature.<n>We define a new task $-$ $textitClaim Relation Classification$ $-$ which requires models to infer the scientific stance toward a cited claim.<n>Our analysis reveals that $63.5$% claims are never reused; only $11.1$% are ever challenged.
  arXiv  Detail & Related papers  (2026-03-17T02:43:36Z)
• HindSight: Evaluating LLM-Generated Research Ideas via Future Impact [5.647839536820347]
  We introduce HindSight, a time-split evaluation framework that measures idea quality by matching generated ideas against real future publications.<n>Using a temporal cutoff$T$, we restrict an idea generation system to pre-$T$ literature, then evaluate its outputs against papers published in the subsequent 30 months.<n>Experiments across 10 AI/ML research topics reveal a striking disconnect: LLM-as-Judge finds no significant difference between retrieval-augmented and vanilla idea generation.
  arXiv  Detail & Related papers  (2026-03-16T11:59:24Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• Can Deep Research Agents Find and Organize? Evaluating the Synthesis Gap with Expert Taxonomies [57.11324429385405]
  We introduce TaxoBench, a diagnostic benchmark derived from 72 computer science surveys.<n>We manually extract expert-authored taxonomy trees containing 3,815 precisely categorized citations as ground truth.<n>Best agent recalls only 20.9% of expert-selected papers, and even with perfect input, the best model achieves only 0.31 ARI in organization.
  arXiv  Detail & Related papers  (2026-01-18T11:57:09Z)
• HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Prediction [14.731720495144112]
  We introduce the novel concept of core citation, which identifies the critical references that go beyond superficial mentions. We propose $textbfHLM-Cite, a $textbfH$ybrid $textbfL$anguage $textbfM$odel workflow for citation prediction. We evaluate HLM-Cite across 19 scientific fields, demonstrating a 17.6% performance improvement comparing SOTA methods.
  arXiv  Detail & Related papers  (2024-10-10T10:46:06Z)
• DAPR: A Benchmark on Document-Aware Passage Retrieval [57.45793782107218]
  We propose and name this task emphDocument-Aware Passage Retrieval (DAPR) While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context. Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
  arXiv  Detail & Related papers  (2023-05-23T10:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.