AMaze: An intuitive benchmark generator for fast prototyping of generalizable agents

• URL: http://arxiv.org/abs/2411.13072v1
• Date: Wed, 20 Nov 2024 06:47:29 GMT
• Title: AMaze: An intuitive benchmark generator for fast prototyping of generalizable agents
• Abstract summary: We introduce AMaze, a novel benchmark generator in which embodied agents must navigate a maze by interpreting visual signs of arbitrary complexities and deceptiveness. As a proof-of-concept, we demonstrate the capabilities of the generator in a simple, fully discrete case with limited deceptiveness.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional approaches to training agents have generally involved a single, deterministic environment of minimal complexity to solve various tasks such as robot locomotion or computer vision. However, agents trained in static environments lack generalization capabilities, limiting their potential in broader scenarios. Thus, recent benchmarks frequently rely on multiple environments, for instance, by providing stochastic noise, simple permutations, or altogether different settings. In practice, such collections result mainly from costly human-designed processes or the liberal use of random number generators. In this work, we introduce AMaze, a novel benchmark generator in which embodied agents must navigate a maze by interpreting visual signs of arbitrary complexities and deceptiveness. This generator promotes human interaction through the easy generation of feature-specific mazes and an intuitive understanding of the resulting agents' strategies. As a proof-of-concept, we demonstrate the capabilities of the generator in a simple, fully discrete case with limited deceptiveness. Agents were trained under three different regimes (one-shot, scaffolding, interactive), and the results showed that the latter two cases outperform direct training in terms of generalization capabilities. Indeed, depending on the combination of generalization metric, training regime, and algorithm, the median gain ranged from 50% to 100% and maximal performance was achieved through interactive training, thereby demonstrating the benefits of a controllable human-in-the-loop benchmark generator.

Related papers

• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks [43.36285223987099]
  We introduce UniClawBench, the first capability-driven benchmark to evaluate proactive agents in dynamic, real-world settings.<n>UniClawBench is built around five executor model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination.<n>Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints.
  arXiv  Detail & Related papers  (2026-07-09T17:59:32Z)
• Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players [87.87842088508553]
  We present our generative multi-agent world model for interactive simulation.<n>Our model improves video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines.
  arXiv  Detail & Related papers  (2026-05-27T17:59:31Z)
• GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoning [54.42973725693]
  We introduce GenAgent, unifying visual understanding and generation through an agentic multimodal model.<n>GenAgent significantly boosts base generator(FLUX.1-dev) performance on GenEval++ and WISE.<n>Our framework demonstrates three key properties: 1) cross-tool generalization to generators with varying capabilities, 2) test-time scaling with consistent improvements across interaction rounds, and 3) task-adaptive reasoning that automatically adjusts to different tasks.
  arXiv  Detail & Related papers  (2026-01-26T14:49:04Z)
• User-Oriented Multi-Turn Dialogue Generation with Tool Use at scale [5.641245411366927]
  We develop a framework for automated task-oriented multi-turn dialogue generation at scale.<n>Our generation pipeline operates as a versatile, plug-and-play module capable of initiating generation from any state.<n>It yields a high-density dataset that reflects the multifaceted demands of real-world human-agent interaction.
  arXiv  Detail & Related papers  (2026-01-13T05:14:09Z)
• Grounded Test-Time Adaptation for LLM Agents [75.62784644919803]
  Large language model (LLM)-based agents struggle to generalize to novel and complex environments.<n>We propose two strategies for adapting LLM agents by leveraging environment-specific information available during deployment.
  arXiv  Detail & Related papers  (2025-11-06T22:24:35Z)
• Emergent Coordination in Multi-Agent Language Models [2.504366738288215]
  We introduce an information-theoretic framework to test whether multi-agent systems show signs of higher-order structure.<n>This information decomposition lets us measure whether dynamical emergence is present in multi-agent LLM systems.<n>We apply our framework to experiments using a simple guessing game without direct agent communication.
  arXiv  Detail & Related papers  (2025-10-05T11:26:41Z)
• MiraGe: Multimodal Discriminative Representation Learning for Generalizable AI-Generated Image Detection [32.662682253295486]
  We propose Multimodal Discriminative Learning for Generalizable AI-generated Image Detection (MiraGegenerator)<n>We apply multimodal prompt learning to further refine these principles into CLIP, leveraging text embeddings as semantic anchors for effective discriminative representation learning.<n>MiraGegenerator achieves state-of-the-art performance, maintaining robustness even against unseen generators like Sora.
  arXiv  Detail & Related papers  (2025-08-03T00:19:18Z)
• Conveying Autonomous Robot Capabilities through Contrasting Behaviour Summaries [8.413049356622201]
  We present an adaptive search method for efficiently generating contrasting behaviour summaries. Our results indicate that adaptive search can efficiently identify informative contrasting scenarios that enable humans to accurately select the better performing agent.
  arXiv  Detail & Related papers  (2023-04-01T18:20:59Z)
• Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors [72.62423312645953]
  Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-based planning and for individual steps. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act in a state-action-based setting. Instead, we combine a mixture of movement primitives with a distribution matching objective to learn versatile behaviors that match the expert's behavior and versatility.
  arXiv  Detail & Related papers  (2022-10-17T16:42:59Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)
• Multi-Agent Imitation Learning with Copulas [102.27052968901894]
  Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions. In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
  arXiv  Detail & Related papers  (2021-07-10T03:49:41Z)
• Evolving Hierarchical Memory-Prediction Machines in Multi-Task Reinforcement Learning [4.030910640265943]
  Behavioural agents must generalize across a variety of environments and objectives over time. We use genetic programming to evolve highly-generalized agents capable of operating in six unique environments from the control literature. We show that emergent hierarchical structure in the evolving programs leads to multi-task agents that succeed by performing a temporal decomposition and encoding of the problem environments in memory.
  arXiv  Detail & Related papers  (2021-06-23T21:34:32Z)
• An Ensemble with Shared Representations Based on Convolutional Networks for Continually Learning Facial Expressions [19.72032908764253]
  Semi-supervised learning through ensemble predictions is an efficient strategy to leverage the high exposure of unlabelled facial expressions during human-robot interactions. Traditional ensemble-based systems are composed of several independent classifiers leading to a high degree of redundancy. We show that our approach is able to continually learn facial expressions through ensemble predictions using unlabelled samples from different data distributions.
  arXiv  Detail & Related papers  (2021-03-05T20:40:52Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning [59.62721526353915]
  Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities. Our method aims to leverage these commonalities by asking the question: What is the expected utility of each agent when only considering a randomly selected sub-group of its observed entities?''
  arXiv  Detail & Related papers  (2020-06-07T18:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.