Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

• URL: http://arxiv.org/abs/2608.16645v3
• Date: Mon, 24 Aug 2026 20:33:03 GMT
• Title: Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies
• Abstract summary: We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature.<n>We ask models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea.
• Score: 5.8079638399400375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.

Related papers

• AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era [55.37558486468993]
  We introduce AgentIdeaBench, a benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration.<n>We report matched Static-Active evaluations for 33 LLMs across 40 densely scored subfields spanning five disciplines.<n>Active exploration reveals considerably more capability headroom, and that headroom is unevenly distributed across models.
  arXiv  Detail & Related papers  (2026-09-07T15:19:13Z)
• Can Large Language Models Forecast What Researchers Study Next? [26.392349686051173]
  Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers.<n>We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF)<n>An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage.
  arXiv  Detail & Related papers  (2026-09-01T05:26:37Z)
• What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation [23.50765366373648]
  We introduce a benchmark that carries a proposal from Literature to Test: the Lit2Test benchmark centers on a six-field contract organized around a falsifying outcome.<n>Built prospectively from 200 real-paper neighborhoods, Lit2Test elicits proposals from four frontier models and compares them through 1,200 pairwise comparisons judged blind in both presentation orders.<n>The protocol audits its own reliability through diagnostic controls and bounded human calibration, with three annotators corroborating the conclusions within explicitly stated reliability bounds.
  arXiv  Detail & Related papers  (2026-08-24T08:17:01Z)
• The Half-Lives of Generative-AI Evidence: A 40-Record Audit, a Claim-Currency Framework, and a Reflexive Case of Frontier-Model-Assisted Research [0.0]
  Generative-AI evaluations can become historical before publication, yet calendar age does not affect every conclusion equally.<n>This paper audits a maximum-variation purposive corpus of 40 empirical records appearing between 18 July 2025 and 17 July 2026.<n>The audit coded publication route, execution timing, model identity, age of the newest named generation or immutable snapshot, same-family supersession and refresh behaviour.
  arXiv  Detail & Related papers  (2026-07-27T05:59:49Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers [0.0]
  ResearchQA is a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers.<n>ResearchQA is designed for citation-grounded evaluation.<n>We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting.
  arXiv  Detail & Related papers  (2026-07-13T04:26:51Z)
• SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones? [51.154921661608675]
  We introduce SoundnessBench, a curated benchmark of 1,099 machine-learning research proposals reconstructed from ICLR submissions.<n>SoundnessBench should be interpreted as a benchmark for recoverable proposal-stage soundness rather than exact prediction of full-paper review outcomes.
  arXiv  Detail & Related papers  (2026-05-28T17:57:37Z)
• Graphs of Research: Citation Evolution Graphs as Supervision for Research Idea Generation [24.044079363031035]
  Graphs of Research (GoR) is a supervised fine-tuning method that extracts a 2-hop reference neighborhood for each seed paper.<n>GoR organizes references into a paper-evolution directed acyclic graph (DAG)<n>We construct an automated extraction pipeline that draws data from five major ML/NLP venues.
  arXiv  Detail & Related papers  (2026-05-14T12:57:56Z)
• GIANTS: Generative Insight Anticipation from Scientific Literature [84.95947892931142]
  We introduce insight anticipation, a generation task in which a model predicts a downstream paper's core insight from its foundational parent papers.<n>We evaluate models using an LM judge that scores similarity between generated and ground-truth insights, and show that these similarity scores correlate with expert human ratings.<n>We present GIANTS-4B, an LM trained via reinforcement learning (RL) to optimize insight anticipation using these similarity scores as a proxy reward.
  arXiv  Detail & Related papers  (2026-04-10T18:13:55Z)
• Learning to Predict Future-Aligned Research Proposals with Language Models [59.79457676644722]
  We build a time-consistent dataset of 17,771 papers from targets and their pre-cutoff citations.<n>To train models, we build a time-consistent dataset of 17,771 papers from targets and their pre-cutoff citations.<n>Across Llama-3.1 and Qwen2.5 models, future-aligned tuning improves future alignment over unaligned baselines.
  arXiv  Detail & Related papers  (2026-03-28T05:41:15Z)
• Alien Science: Sampling Coherent but Cognitively Unavailable Research Directions from Idea Atoms [53.907293349123506]
  Large language models often fail at producing ideas that are both coherent and non-obvious to the current community.<n>We formalize this gap through cognitive availability, the likelihood that a research direction would be naturally proposed by a typical researcher.<n>We learn two complementary models: a coherence model that scores whether a set of atoms constitutes a viable direction, and an availability model that scores how likely that direction is to be generated.
  arXiv  Detail & Related papers  (2026-03-01T13:05:19Z)
• A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks [72.7373468905418]
  We develop an open-source toolkit OpenBackdoor to foster the implementations and evaluations of textual backdoor learning. We also propose CUBE, a simple yet strong clustering-based defense baseline.
  arXiv  Detail & Related papers  (2022-06-17T02:29:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.