RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases

• URL: http://arxiv.org/abs/2609.10092v1
• Date: Wed, 09 Sep 2026 12:15:14 GMT
• Title: RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases
• Abstract summary: Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate.<n>We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes.<n>At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions.
• Score: 58.92094935632493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate because reviews and research ideas lack uniquely verifiable outcomes. We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes. At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions. Search generally helps, but all four diagnostic models perform worse than an exact-count exponentially weighted moving average (EWMA) baseline in compositional accuracy. We identify two linked bottlenecks. Under cumulative-history access, State carry-forward outperforms direct Forecast for all four diagnostic models; frozen-evidence replay links a shared component of this reversal to Forecast-oriented policies retrieving a smaller share of recent evidence. Even with exact historical activity, future-specific updating remains limited, with only GPT-5.5 plus reopened Search slightly surpassing EWMA. Fine-tuning on realised outcomes improves Qwen3-4B's forecast Spearman correlation by 0.105 on held-out fields at later origins, with gains also on change-rich episodes.

Related papers

• Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures [11.365690419023425]
  Existing root-cause attribution methods suffer from low diagnostic accuracy as execution traces grow larger.<n>Continual Search nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence.<n>MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks.
  arXiv  Detail & Related papers  (2026-09-11T19:29:21Z)
• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Can Large Language Models Forecast What Researchers Study Next? [26.392349686051173]
  Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers.<n>We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF)<n>An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage.
  arXiv  Detail & Related papers  (2026-09-01T05:26:37Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods [4.424680117690851]
  We prompt GPT-5.1, Gemini 3 Pro, and DeepSeek-V3.2 with an LLM-extracted research question from each of 1,000 recent arXiv computer-science papers.<n>Since we provide only the research question, the differences we measure reflect initial suggestions and not how optimal those suggestions are.<n>The strongest imbalance appears in provider choice, with Jensen-Shannon divergence about 3-5x larger than any other taxonomy dimension.
  arXiv  Detail & Related papers  (2026-06-15T14:53:30Z)
• ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment [22.38716170940567]
  We introduce ForeSci, a benchmark for evaluating whether research agents can make forward-looking judgements from historical evidence.<n>ForeSci contains 500 tasks across four fast-moving AI domains and four decision families.<n>We show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family.
  arXiv  Detail & Related papers  (2026-05-30T09:41:26Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• Data-driven Lake Water Quality Forecasting for Time Series with Missing Data using Machine Learning [4.683806391173103]
  Volunteer-led lake monitoring yields irregular, seasonal time series with many gaps arising from ice cover, weather-related access constraints, and occasional human errors.<n>We study Secchi Disk Depth (SDD) forecasting on a 30-lake, data-rich subset drawn from three decades of in situ records collected across Maine lakes.
  arXiv  Detail & Related papers  (2026-01-21T22:20:26Z)
• Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination [77.69093448529455]
  We present an empirical study using an infinitely scalable framework to synthesize research-level QA directly from arXiv papers.<n>We evaluate a lack of significant performance decay near knowledge cutoff dates for models of various sizes, developers, and release dates.<n>We hypothesize that the multi-step reasoning required by our synthesis pipeline offered additional complexity that goes deeper than shallow memorization.
  arXiv  Detail & Related papers  (2025-08-26T16:41:37Z)
• Ensembled Prediction Intervals for Causal Outcomes Under Hidden Confounding [49.1865229301561]
  We present a simple approach to partial identification using existing causal sensitivity models and show empirically that Caus-Modens gives tighter outcome intervals. The last of our three diverse benchmarks is a novel usage of GPT-4 for observational experiments with unknown but probeable ground truth.
  arXiv  Detail & Related papers  (2023-06-15T21:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.