Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design

• URL: http://arxiv.org/abs/2605.15871v1
• Date: Fri, 15 May 2026 11:40:41 GMT
• Title: Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design
• Abstract summary: We introduce a dual-framework approach: AIRA-Compose for high-level architecture search, and AIRA-Design for low-level mechanistic implementation.<n>AirA-Compose uses 11 agents to explore fundamental computational primitives under a 24-hour budget.<n>AirAformers (Transformer-based) and AIRAhybrids (Transformer-Mamba) consistently outperform Llama 3.2 and Composer-found baselines.
• Score: 9.382770582572784
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Toward recursive self-improvement, we investigate LLM agents autonomously designing foundation models beyond standard Transformers. We introduce a dual-framework approach: AIRA-Compose for high-level architecture search, and AIRA-Design for low-level mechanistic implementation. AIRA-Compose uses 11 agents to explore fundamental computational primitives under a 24-hour budget. Agents evaluate million-parameter candidates, extrapolating top designs to 350M, 1B, and 3B scales. This yields 14 architectures across two families: AIRAformers (Transformer-based) and AIRAhybrids (Transformer-Mamba). Pre-trained at 1B scale, these consistently outperform Llama 3.2 and Composer-found baselines. On downstream tasks, AIRAformer-D and AIRAhybrid-D improve accuracy by 2.4% and 3.8% over Llama 3.2. Furthermore, AIRA-Compose finds models with highly efficient scaling frontiers: AIRAformer-C scales 54% and 71% faster than Llama 3.2 and Composer's best Transformer, while AIRAhybrid-C outscales Nemotron-2 by 23% and Composer's best hybrid by 37%. AIRA-Design tasks 20 agents with writing novel attention mechanisms for long-range dependencies and high-performing training scripts. On the Long Range Arena benchmark, agent-designed architectures reach within 2.3% and 2.6% of human state-of-the-art on document matching and text classification. On the Autoresearch benchmark, Greedy Opus 4.5 achieves 0.968 validation bits-per-byte under a fixed time budget, surpassing the published minimum. Together, these frameworks show AI agents can autonomously discover architectures and algorithmic optimizations matching or surpassing hand-designed baselines. This establishes a powerful paradigm for discovering next-generation foundation models, marking a clear step toward recursive self-improvement.

Related papers

• Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1 [4.486629113012585]
  Recent progress on ARC-AGI-1 has come from two regimes: heavy test-time compute over frontier models, or benchmark-specific training in which small models are fine-tuned on ARC data.<n>We study what is recoverable through architecture alone, building agentic harnesses that decompose pattern-discovery and program-synthesis stages explicitly.
  arXiv  Detail & Related papers  (2026-07-07T19:49:35Z)
• Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent [98.33217711280383]
  We introduce Agents-A1, a 35B Mixture-of-Experts Agentic Model that reaches trillion- parameter-level performance by scaling the agent horizon.<n>We build a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes.<n>Agents-A1 achieves strong and broad performance for long-horizon agent benchmarks.
  arXiv  Detail & Related papers  (2026-06-29T17:50:54Z)
• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models [63.96252564303143]
  Embodied-R1.5 is a unified Embodied Foundation Model (EFM) that integrates comprehensive embodied reasoning capabilities.<n>We build a large-scale data system of over 15B tokens, and design a multi-task balanced RL recipe.<n>With only 8B parameters, Embodied-R1.5 SOTA on 16 out of 24 embodied VLM benchmarks, surpassing leading models like Gemini-Robotics-ER-1.5 and GPT-5.4.
  arXiv  Detail & Related papers  (2026-06-09T18:07:50Z)
• PersonalAI 2.0: Enhancing knowledge graph traversal/retrieval with planning mechanism for Personalized LLM Agents [53.369140518949656]
  We introduce PersonalAI 2.0 (PAI-2), a novel framework to enhance large language model (LLM) based systems through integration of external knowledge graphs (KG)<n>The proposed approach addresses key limitations of existing Retrieval-Augmented Generation (GraphRAG) methods by incorporating a dynamic, multistage query processing pipeline.<n>We show that use of graph algorithms (e.g. BeamSearch, WaterCircles) gain superior results compared to standard flatten retriever on average 6%, while enabled search plan enhancement mechanism gain 18% boost compared to disabled one by LLM-as-a-Judge across six datasets.
  arXiv  Detail & Related papers  (2026-05-13T13:06:30Z)
• RoboPhD: Evolving Diverse Complex Agents Under Tight Evaluation Budgets [0.2126018197183873]
  We present the first systematic comparison of three optimization paradigms -- Elo tournament selection, GEPA, and greedy hill-climbing.<n>RoboPhD introduces validation-free evolution: instead of splitting the budget between training and validation, it uses Elo competition on training data to simultaneously evaluate agents and drive evolution.<n>We release RoboPhD as a versatile toolkit under the MIT license with a simple optimize_anything() API for evolving diverse complex agents.
  arXiv  Detail & Related papers  (2026-04-06T01:39:13Z)
• WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning [82.12501258760814]
  Large Language Model(LLM)-based agents have shown strong capabilities in web information seeking.<n>Plan anchor is where the first reasoning step disproportionately impacts downstream behavior in long-horizon web reasoning tasks.<n>We propose Anchor-GRPO, a two-stage RL framework that decouples planning and execution.
  arXiv  Detail & Related papers  (2026-01-06T16:36:40Z)
• ROAD: Reflective Optimization via Automated Debugging for Zero-Shot Agent Alignment [1.6968020497268546]
  ROAD is a novel framework that treats optimization as a dynamic debug investigation rather than a search.<n>Road is highly sample-efficient, achieving a 5.6 percent increase in success rate and a 3.8 percent increase in search accuracy.<n>These findings suggest that mimicking the human engineering loop of failure analysis and patching offers a viable, data-efficient alternative to resource-intensive training.
  arXiv  Detail & Related papers  (2025-12-30T07:31:34Z)
• AlphaApollo: Orchestrating Foundation Models and Professional Tools into a Self-Evolving System for Deep Agentic Reasoning [110.57865233597762]
  We present AlphaApollo, a self-evolving agentic reasoning system.<n>It aims to address two bottlenecks in foundation model (FM) reasoning-limited model-intrinsic capacity and unreliable test-time computation.<n>AlphaApollo orchestrates multiple models with professional tools to enable deliberate, verifiable reasoning.
  arXiv  Detail & Related papers  (2025-10-05T15:42:24Z)
• Investigating the Potential of Large Language Model-Based Router Multi-Agent Architectures for Foundation Design Automation: A Task Classification and Expert Selection Study [0.0]
  Three approaches were evaluated: single-agent processing, multi-agent designer-checker architecture, and router-based expert selection.<n>Performance assessment utilized baseline models including DeepSeek R1, ChatGPT 4 Turbo, Grok 3, and Gemini 2.5 Pro.<n>Results establish router-based multi-agent systems as optimal for foundation design automation while maintaining professional documentation standards.
  arXiv  Detail & Related papers  (2025-06-13T23:45:24Z)
• Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training [73.90260246781435]
  We present Lory, the first approach that scales such architectures to autoregressive language model pre-training. We show significant performance gains over parameter-matched dense models on both perplexity and a variety of downstream tasks. Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing.
  arXiv  Detail & Related papers  (2024-05-06T03:06:33Z)
• An Empirical Study of Multimodal Model Merging [148.48412442848795]
  Model merging is a technique that fuses multiple models trained on different tasks to generate a multi-task solution. We conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture. We propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes.
  arXiv  Detail & Related papers  (2023-04-28T15:43:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.