AIRA_2: Overcoming Bottlenecks in AI Research Agents

• URL: http://arxiv.org/abs/2603.26499v1
• Date: Fri, 27 Mar 2026 15:02:43 GMT
• Title: AIRA_2: Overcoming Bottlenecks in AI Research Agents
• Abstract summary: Existing research has identified three structural performance bottlenecks in AI research agents.<n>We introduce AIRA$$, which addresses these bottlenecks through three architectural choices.<n>On MLE-bench-30, AIRA$$ achieves a mean Percentile Rank of 71.8% at 24 hours - surpassing the previous best of 69.9%.
• Score: 31.269750755536247
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing research has identified three structural performance bottlenecks in AI research agents: (1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes performance to degrade over extended search horizons; and (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance. We introduce AIRA$_2$, which addresses these bottlenecks through three architectural choices: an asynchronous multi-GPU worker pool that increases experiment throughput linearly; a Hidden Consistent Evaluation protocol that delivers a reliable evaluation signal; and ReAct agents that dynamically scope their actions and debug interactively. On MLE-bench-30, AIRA$_2$ achieves a mean Percentile Rank of 71.8% at 24 hours - surpassing the previous best of 69.9% - and steadily improves to 76.0% at 72 hours. Ablation studies reveal that each component is necessary and that the "overfitting" reported in prior work was driven by evaluation noise rather than true data memorization.

Related papers

• SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search [60.29041788383166]
  We present SimpleSearch-VL, an efficient, reliable, and practical framework for multimodal agentic search.<n>SimpleSearch-VL improves the agent's own search-and-verification process rather than scaling data, tools, or auxiliary model components.
  arXiv  Detail & Related papers  (2026-06-30T11:22:24Z)
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• SPOQ: Specialist Orchestrated Queuing for Multi-Agent Software Engineering [2.0769172070951067]
  Multi-agent AI systems show promise for automating software engineering tasks.<n>Existing approaches suffer from coordination overhead, quality control gaps, and limited human oversight.<n>We introduce SPOQ (Specialist Orchestrated Queuing), a methodology combining three innovations.
  arXiv  Detail & Related papers  (2026-06-02T03:59:56Z)
• FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics [24.125726163497742]
  FML-Bench is a benchmark of 18 fundamental ML research tasks across 10 domains.<n>It separates agent strategy from execution infrastructure and defines 12 process-level behavioral metrics.<n>We find greedy search tends to be more effective when opportunities are dense, while tree-search and evolutionary strategies tend to be more effective when opportunities are sparse.
  arXiv  Detail & Related papers  (2026-05-17T10:30:38Z)
• PersonalAI 2.0: Enhancing knowledge graph traversal/retrieval with planning mechanism for Personalized LLM Agents [53.369140518949656]
  We introduce PersonalAI 2.0 (PAI-2), a novel framework to enhance large language model (LLM) based systems through integration of external knowledge graphs (KG)<n>The proposed approach addresses key limitations of existing Retrieval-Augmented Generation (GraphRAG) methods by incorporating a dynamic, multistage query processing pipeline.<n>We show that use of graph algorithms (e.g. BeamSearch, WaterCircles) gain superior results compared to standard flatten retriever on average 6%, while enabled search plan enhancement mechanism gain 18% boost compared to disabled one by LLM-as-a-Judge across six datasets.
  arXiv  Detail & Related papers  (2026-05-13T13:06:30Z)
• Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis [68.28714988482703]
  Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs)<n>We present a study revealing that general-domain PRMs struggle to supervise data analysis agents.<n>We introduce DataPRM, a novel environment-aware generative process reward model.
  arXiv  Detail & Related papers  (2026-04-27T09:00:30Z)
• Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation [95.89924101984566]
  We introduce OptimusVLA, a dual-memory VLA framework with Global Prior Memory (GPM) and Local Consistency Memory (LCM)<n>GPM replaces Gaussian noise with task-level priors retrieved from semantically similar trajectories.<n>LCM injects a learned consistency constraint that enforces temporal coherence and smoothness of trajectory.
  arXiv  Detail & Related papers  (2026-02-22T15:39:34Z)
• AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering [52.67783579040657]
  AceGRPO is a machine learning system that prioritizes tasks at the agent's learning frontier to maximize learning efficiency.<n>Our trained Ace-30B model achieves a 100% valid submission rate on MLE-Bench-Lite, approaches the performance of proprietary frontier models, and outperforms larger open-source baselines.
  arXiv  Detail & Related papers  (2026-02-08T10:55:03Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• AI, Metacognition, and the Verification Bottleneck: A Three-Wave Longitudinal Study of Human Problem-Solving [0.0]
  This pilot study tracked how generative AI reshapes problem-solving over six months in an academic setting.<n>Results generalize primarily to early-adopter, academically affiliated populations.
  arXiv  Detail & Related papers  (2026-01-21T15:49:04Z)
• LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum [73.82125917416067]
  LACONIC is a family of learned sparse retrievers based on the Llama-3 architecture.<n>The 8B variant achieves a state-of-the-art 60.2 nDCG on the MTEB Retrieval benchmark, ranking 15th on the leaderboard.
  arXiv  Detail & Related papers  (2026-01-04T22:42:20Z)
• HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation [21.08814504507274]
  suboptimal search behaviors exist widely, such as over-search and under-search.<n>Current training methods, which typically rely on outcome-based rewards in a RL framework, lack the fine-grained control needed to address these inefficiencies.<n>We introduce HiPRAG, a training methodology that incorporates a fine-grained, knowledge-grounded process reward into the RL training.
  arXiv  Detail & Related papers  (2025-10-09T05:13:10Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective [3.2771631221674333]
  We leverage task-specific data augmentations throughout the training, generation, and scoring phases.<n>We employ a depth-first search algorithm to generate diverse, high-probability candidate solutions.<n>Our method achieves a score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set.
  arXiv  Detail & Related papers  (2025-05-08T11:17:10Z)
• ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention [48.697458429460184]
  Two factors, information bottleneck sensitivity and inconsistency between different attention topologies, could affect the performance of the Sparse Transformer. This paper proposes a well-designed model named ERNIE-Sparse. It consists of two distinctive parts: (i) Hierarchical Sparse Transformer (HST) to sequentially unify local and global information, and (ii) Self-Attention Regularization (SAR) to minimize the distance for transformers with different attention topologies.
  arXiv  Detail & Related papers  (2022-03-23T08:47:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.