ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment

• URL: http://arxiv.org/abs/2606.00644v2
• Date: Thu, 04 Jun 2026 03:19:03 GMT
• Title: ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment
• Abstract summary: We introduce ForeSci, a benchmark for evaluating whether research agents can make forward-looking judgements from historical evidence.<n>ForeSci contains 500 tasks across four fast-moving AI domains and four decision families.<n>We show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family.
• Score: 22.38716170940567
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, and answer-generation backbones are selected to precede the task cutoffs. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Results show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. ForeSci turns forward-looking AI research judgement into a controlled benchmark for evaluating research agents as decision-making systems.

Related papers

• RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases [58.92094935632493]
  Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate.<n>We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes.<n>At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions.
  arXiv  Detail & Related papers  (2026-09-09T12:15:14Z)
• AutoResearch: Insight In, Hallucination Out [18.44583561825209]
  We introduce AutoResearch, a two-stage system that connects Idea Generation and Idea Execution.<n>In Idea Generation, AutoResearch continuously integrates emerging research signals with accumulated domain knowledge.<n>In Idea Execution, coordinated agents decompose these plans into experiments, iteratively implement and diagnose them, and employ independent evidence-based review.
  arXiv  Detail & Related papers  (2026-08-18T15:38:26Z)
• AREX: Towards a Recursively Self-Improving Agent for Deep Research [75.96468303743958]
  We introduce AREX, a family of Recursively Self-Improving deep research agents.<n>AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer.<n>We train AREX on verified synthetic tasks and high-quality trajectories through agentic mid-training and long-horizon reinforcement learning.
  arXiv  Detail & Related papers  (2026-07-23T16:05:46Z)
• Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions [24.28151698249644]
  Large language models (LLMs) and agentic AI systems have emerged as promising solutions.<n>This survey systematically reviews 74 studies and makes four contributions.
  arXiv  Detail & Related papers  (2026-07-03T11:42:29Z)
• AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration [175.74514061083195]
  We present AutoResearchClaw, a multi-agent autonomous research pipeline built on five mechanisms.<n>On ARC-Bench, a 25-topic experiment-stage benchmark, AutoResearchClaw outperforms AI Scientist v2 by 54.7%.
  arXiv  Detail & Related papers  (2026-05-19T15:49:51Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• The Last Human-Written Paper: Agent-Native Research Artifacts [106.47848184955576]
  We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package.<n>Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste.
  arXiv  Detail & Related papers  (2026-04-27T16:23:09Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification [71.98473277917962]
  Recent advances in Deep Research Agents (DRAs) are transforming automated knowledge discovery and problem-solving.<n>We propose an alternative paradigm: self-evolving the agent's ability by iteratively verifying the policy model's outputs, guided by meticulously crafted rubrics.<n>We present DeepVerifier, a rubrics-based outcome reward verifier that leverages the asymmetry of verification.
  arXiv  Detail & Related papers  (2026-01-22T09:47:31Z)
• Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents [3.0745879700441385]
  PROBE decomposes proactivity as a pipeline of three core capabilities.<n>We find that the best end-to-end performance of 40% is achieved by both GPT-5 and Claude Opus-4.1.
  arXiv  Detail & Related papers  (2025-10-22T17:00:45Z)
• Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them [23.986035712600657]
  We propose a reasoning-driven pipeline to study effective reasoning behavior patterns in agentic search.<n>We identify four beneficial reasoning behaviors: Information Verification, Authority Evaluation, Adaptive Search, and Error Recovery.<n>We show that behavior priming yields over 35% gains in Llama3.2-3B and Qwen3-1.7B compared to directly training agentic search models with RL.
  arXiv  Detail & Related papers  (2025-10-08T00:20:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.