Can Retrievers Find the Same Paper from Different Aspects? A Multi-Aspect Full-Paper Scientific Retrieval Benchmark

• URL: http://arxiv.org/abs/2608.15624v1
• Date: Sun, 16 Aug 2026 08:41:23 GMT
• Title: Can Retrievers Find the Same Paper from Different Aspects? A Multi-Aspect Full-Paper Scientific Retrieval Benchmark
• Abstract summary: MAPLE is an expert-validated benchmark for multi-aspect, full-paper retrieval.<n>We propose MAPLE- Synth, a retrieval-based in-context learning pipeline.<n>Experiments reveal a substantial gap between retrieving a paper from any one aspect and retrieving it from all aspects.
• Score: 13.55197884375559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific papers contain multiple searchable facets such as background, methods. However, many paper retrieval benchmarks merely evaluate individual query-paper relevance, while overlooking other facets of the same paper. To bridge this gap, we introduce MAPLE, an expert-validated benchmark for multi-aspect, full-paper retrieval that evaluates whether retrievers can consistently recover the same paper from queries targeting its motivation, method, and experimental findings. MAPLE contains 2,095 queries about recent ML and NLP papers, grounded in both textual and multimodal content. We further propose MAPLE-Synth, a retrieval-based in-context learning pipeline that leverages OpenReview discussions and human-written query exemplars to generate realistic queries reflecting researchers' interests in different aspects of a paper. Our expert validation shows that these queries are comparable in realism to human-written queries and highly relevant to the target papers. Experiments across lexical, scientific-domain, general-purpose text, and multimodal retrievers reveal a substantial gap between retrieving a paper from any one aspect and retrieving it from all aspects: the strongest model achieves 98.1% AnyAspect@20 but only 15.7% AllAspect@20. Experiment/result queries and table-referenced queries are particularly difficult across retrievers. Although multi-chunk aggregation improves multi-aspect paper retrieval, considerable failures persist. MAPLE provides a testbed for evaluating and developing retrievers that represent scientific papers more comprehensively.

Related papers

• PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers [52.97586643334384]
  We introduce PaperScope, a benchmark for agentic deep research.<n>It is built on a knowledge graph of over 2,000 AI papers spanning three years.<n>It integrates semantically related key information nodes and employs optimized random-walk article selector.
  arXiv  Detail & Related papers  (2026-04-13T11:07:08Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrieval [36.38599923075882]
  DIVER is a retrieval pipeline designed for reasoning-intensive information retrieval.<n>It consists of four components: the document preprocessing stage, the query expansion stage, the retrieval stage and the reranking stage.<n>On the BRIGHT benchmark, DIVER achieves state-of-the-art nDCG@10 scores of 45.8 overall and 28.9 on original queries, consistently outperforming competitive reasoning-aware models.
  arXiv  Detail & Related papers  (2025-08-11T13:57:49Z)
• Chain of Retrieval: Multi-Aspect Iterative Search Expansion and Post-Order Search Aggregation for Full Paper Retrieval [68.71038700559195]
  Chain of Retrieval(COR) is a novel iterative framework for full-paper retrieval.<n>We present SCIBENCH, a benchmark providing both complete and segmented contexts of full papers for queries and candidates.
  arXiv  Detail & Related papers  (2025-07-14T08:41:53Z)
• MMDocIR: Benchmarking Multimodal Retrieval for Long Documents [44.02536756069517]
  This work introduces a new benchmark, named MMDocIR, that encompasses two distinct tasks: page-level and layout-level retrieval.<n>The MMDocIR benchmark comprises a rich dataset featuring 1,685 questions annotated by experts and 173,843 questions with bootstrapped labels.
  arXiv  Detail & Related papers  (2025-01-15T14:30:13Z)
• Open-World Evaluation for Retrieving Diverse Perspectives [39.22331280176582]
  We curate a Benchmark for Retrieval Diversity for Subjective questions (BERDS)<n>Each example consists of a question and diverse perspectives associated with the question.<n>We build a language model-based automatic evaluator that decides whether each retrieved document contains a perspective.
  arXiv  Detail & Related papers  (2024-09-26T17:52:57Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness [56.42192735214931]
  retrievers are expected to not only rely on the semantic relevance between the documents and the queries but also recognize the nuanced intents or perspectives behind a user query. In this work, we study whether retrievers can recognize and respond to different perspectives of the queries. We show that current retrievers have limited awareness of subtly different perspectives in queries and can also be biased toward certain perspectives.
  arXiv  Detail & Related papers  (2024-05-04T17:10:00Z)
• Multi-View Document Representation Learning for Open-Domain Dense Retrieval [87.11836738011007]
  This paper proposes a multi-view document representation learning framework. It aims to produce multi-view embeddings to represent documents and enforce them to align with different queries. Experiments show our method outperforms recent works and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2022-03-16T03:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.