ARE: Scaling Up Agent Environments and Evaluations

• URL: http://arxiv.org/abs/2509.17158v1
• Date: Sun, 21 Sep 2025 16:59:45 GMT
• Title: ARE: Scaling Up Agent Environments and Evaluations
• Authors: Pierre Andrews, Amine Benhalloum, Gerard Moreno-Torres Bertran, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Romain Froger, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Grégoire Mialon, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Thomas Scialom, Vladislav Vorotilov, Mengjue Wang, Ian Yu,
• Abstract summary: We introduce Meta Agents Research Environments (ARE), a research platform for scalable creation of environments.<n>ARE provides simple abstractions to build complex and diverse environments, each with their own rules, tools, content, and verifiers.<n>We also propose Gaia2, a benchmark built in ARE and designed to measure general agent capabilities.
• Score: 22.98982051873728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Meta Agents Research Environments (ARE), a research platform for scalable creation of environments, integration of synthetic or real applications, and execution of agentic orchestrations. ARE provides simple abstractions to build complex and diverse environments, each with their own rules, tools, content, and verifiers, helping to bridge the gap between model development and real-world deployment. We also propose Gaia2, a benchmark built in ARE and designed to measure general agent capabilities. Beyond search and execution, Gaia2 requires agents to handle ambiguities and noise, adapt to dynamic environments, collaborate with other agents, and operate under temporal constraints. Unlike prior benchmarks, Gaia2 runs asynchronously, surfacing new failure modes that are invisible in static settings. Our experiments show that no system dominates across the intelligence spectrum: stronger reasoning often comes at the cost of efficiency, and budget scaling curves plateau, highlighting the need for new architectures and adaptive compute strategies. Perhaps more importantly, ARE abstractions enable continuous extension of Gaia2 to other environments, empowering the community to rapidly create new benchmarks tailored to their domains. In AI's second half, progress increasingly depends on defining meaningful tasks and robust evaluations to drive frontier capabilities forward.

Related papers

• Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments [22.98982051873728]
  We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments.<n>Gaia2 introduces scenarios where environments evolve independently of agent actions.<n>Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform.
  arXiv  Detail & Related papers  (2026-02-12T13:58:27Z)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning [62.499592503950026]
  Large language model (LLM) have empowered autonomous agents to perform complex tasks that require multi-turn interactions with tools and environments.<n>We propose Agent World Model (AWM), a fully synthetic environment generation pipeline.<n>We scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets.
  arXiv  Detail & Related papers  (2026-02-10T18:55:41Z)
• What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding [50.35012849818872]
  Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks.<n>We propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding.<n>Our experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment.
  arXiv  Detail & Related papers  (2026-01-14T14:09:11Z)
• Grounded Test-Time Adaptation for LLM Agents [75.62784644919803]
  Large language model (LLM)-based agents struggle to generalize to novel and complex environments.<n>We propose two strategies for adapting LLM agents by leveraging environment-specific information available during deployment.
  arXiv  Detail & Related papers  (2025-11-06T22:24:35Z)
• Towards General Agentic Intelligence via Environment Scaling [78.66355092082253]
  Advanced agentic intelligence is a prerequisite for deploying Large Language Models in real-world applications.<n>We design a scalable framework that automatically constructs heterogeneous environments that are fully simulated.<n>Experiments on agentic benchmarks, tau-bench, tau2-Bench, and ACEBench, demonstrate that our trained model, AgentScaler, significantly enhances the function-calling capability of models.
  arXiv  Detail & Related papers  (2025-09-16T17:57:20Z)
• CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale [4.464959191643012]
  We introduce CREW-Wildfire, an open-source benchmark designed to evaluate next-generation multi-agent Agentic AI frameworks.<n> CREW-Wildfire offers procedurally generated wildfire response scenarios featuring large maps, heterogeneous agents, partial observability, dynamics, and long-horizon planning objectives.<n>We implement and evaluate several state-of-the-art LLM-based multi-agent Agentic AI frameworks, uncovering significant performance gaps.
  arXiv  Detail & Related papers  (2025-07-07T16:33:42Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments [93.94020724735199]
  HAZARD consists of three unexpected disaster scenarios, including fire, flood, and wind. This benchmark enables us to evaluate autonomous agents' decision-making capabilities across various pipelines.
  arXiv  Detail & Related papers  (2024-01-23T18:59:43Z)
• AI planning in the imagination: High-level planning on learned abstract search spaces [68.75684174531962]
  We propose a new method, called PiZero, that gives an agent the ability to plan in an abstract search space that the agent learns during training. We evaluate our method on multiple domains, including the traveling salesman problem, Sokoban, 2048, the facility location problem, and Pacman.
  arXiv  Detail & Related papers  (2023-08-16T22:47:16Z)
• WebArena: A Realistic Web Environment for Building Autonomous Agents [92.3291458543633]
  We build an environment for language-guided agents that is highly realistic and reproducible. We focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains. We release a set of benchmark tasks focusing on evaluating the functional correctness of task completions.
  arXiv  Detail & Related papers  (2023-07-25T22:59:32Z)
• Novelty Accommodating Multi-Agent Planning in High Fidelity Simulated Open World [7.821603097781892]
  We address the challenge that arises when unexpected phenomena, termed textitnovelties, emerge within the environment.<n>The introduction of novelties into the environment can lead to inaccuracies within the planner's internal model, rendering previously generated plans obsolete.<n>We propose a general purpose AI agent framework designed to detect, characterize, and adapt to support concurrent actions and external scheduling.
  arXiv  Detail & Related papers  (2023-06-22T03:44:04Z)
• BEHAVIOR in Habitat 2.0: Simulator-Independent Logical Task Description for Benchmarking Embodied AI Agents [31.499374840833124]
  We bring a subset of BEHAVIOR activities into Habitat 2.0 to benefit from its fast simulation speed. Inspired by the catalyzing effect that benchmarks have played in the AI fields, the community is looking for new benchmarks for embodied AI.
  arXiv  Detail & Related papers  (2022-06-13T21:37:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.