Gym-V: A Unified Vision Environment System for Agentic Vision Research

• URL: http://arxiv.org/abs/2603.15432v2
• Date: Tue, 17 Mar 2026 17:07:16 GMT
• Title: Gym-V: A Unified Vision Environment System for Agentic Vision Research
• Abstract summary: Gym-V is a unified platform of 179 procedurally generated visual environments across 10 domains with controllable difficulty.<n>We find that observation scaffolding is more decisive for training success than the choice of RL algorithm.<n>Cross-domain transfer experiments show that training on diverse task generalizes broadly while narrow training can cause negative transfer.
• Score: 51.62136909074774
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As agentic systems increasingly rely on reinforcement learning from verifiable rewards, standardized ``gym'' infrastructure has become essential for rapid iteration, reproducibility, and fair comparison. Vision agents lack such infrastructure, limiting systematic study of what drives their learning and where current models fall short. We introduce \textbf{Gym-V}, a unified platform of 179 procedurally generated visual environments across 10 domains with controllable difficulty, enabling controlled experiments that were previously infeasible across fragmented toolkits. Using it, we find that observation scaffolding is more decisive for training success than the choice of RL algorithm, with captions and game rules determining whether learning succeeds at all. Cross-domain transfer experiments further show that training on diverse task categories generalizes broadly while narrow training can cause negative transfer, with multi-turn interaction amplifying all of these effects. Gym-V is released as a convenient foundation for training environments and evaluation toolkits, aiming to accelerate future research on agentic VLMs.

Related papers

• Towards Generalizable Visually Grounded Exploration of Household Devices [74.66117923766632]
  Key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration.<n>We introduce VGEBench, a benchmark designed to evaluate the generalizable visually grounded exploration capabilities of Vision-Language Models.<n>Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction.
  arXiv  Detail & Related papers  (2026-09-01T07:42:45Z)
• ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning [53.17987412112712]
  Reinforcement Learning (RL) has demonstrated significant potential for improving Vision-Language-Action (VLA) models on complex manipulation tasks.<n>In this work, we first investigate the exploration stagnation bottleneck in current VLA-RL frameworks.<n>We introduce RL Exploration Token (ExToken), a simple yet general framework that condition VLA policies on discrete behavioral priors.
  arXiv  Detail & Related papers  (2026-07-14T16:04:41Z)
• See First, Answer Later: Visual Evidence Pre-Alignment via Sufficiency-Driven RL [19.682815357013453]
  Multimodal large language models (MLLMs) integrate strong text reasoning with visual inputs, yet their responses can be inconsistent with the underlying images.<n>We introduce Visual Evidence Pre-Alignment (VEPA), an intermediate stage between pretraining and post-training to optimize question-conditioned visual evidence descriptions.
  arXiv  Detail & Related papers  (2026-06-16T08:45:24Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks [76.00315860962885]
  We propose CLASP (CLIP-guided Adaptable Self-suPervised learning), a novel framework for unsupervised pre-training in human-centric visual tasks.<n> CLASP leverages the powerful vision-language model CLIP to generate both low-level (e.g., body parts) and high-level (e.g., attributes) semantic pseudo-labels.<n>MoE dynamically adapts feature extraction based on task-specific prompts, mitigating potential feature conflicts and enhancing transferability.
  arXiv  Detail & Related papers  (2026-01-19T15:19:28Z)
• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning [59.88543114325153]
  We introduce the Seeing-to-Experiencing framework to scale the capability of navigation foundation models with reinforcement learning.<n>S2E combines the strengths of pre-training on videos and post-training through RL.<n>We establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3DGS reconstructions of real-world scenes.
  arXiv  Detail & Related papers  (2025-07-29T17:26:10Z)
• VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection [47.259066449806866]
  VisTA is a new reinforcement learning framework that empowers visual agents to dynamically explore, select, and combine tools from a diverse library based on empirical performance.<n>We show that VisTA achieves substantial performance gains over training-free baselines.<n>These results highlight VisTA's ability to enhance generalization, adaptively utilize diverse tools, and pave the way for flexible, experience-driven visual reasoning systems.
  arXiv  Detail & Related papers  (2025-05-26T17:59:17Z)
• Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning [3.8309622155866583]
  We introduce the Sliding Puzzles Gym (SPGym), a novel benchmark that transforms the classic 8-tile puzzle into a visual reinforcement learning task with images drawn from arbitrarily large datasets.<n>SPGym's key innovation lies in its ability to precisely control representation learning complexity through adjustable grid sizes and image pools.
  arXiv  Detail & Related papers  (2024-10-17T21:23:03Z)
• Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks [139.3768582233067]
  Battle of the Backbones (BoB) is a benchmarking tool for neural network based computer vision systems. We find that vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular. In apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive.
  arXiv  Detail & Related papers  (2023-10-30T18:23:58Z)
• VIBR: Learning View-Invariant Value Functions for Robust Visual Control [3.2307366446033945]
  VIBR (View-Invariant Bellman Residuals) is a method that combines multi-view training and invariant prediction to reduce out-of-distribution gap for RL based visuomotor control. We show that VIBR outperforms existing methods on complex visuo-motor control environment with high visual perturbation.
  arXiv  Detail & Related papers  (2023-06-14T14:37:34Z)
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models [115.49214555402567]
  Pre-trained weights often boost a wide range of downstream tasks including classification, detection, and segmentation. Recent studies suggest that pre-training benefits from gigantic model capacity. In this paper, we examine supervised and self-supervised pre-trained models through the lens of the lottery ticket hypothesis (LTH)
  arXiv  Detail & Related papers  (2020-12-12T21:53:55Z)
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation [115.4071729927011]
  We study the effects of using mid-level visual representations as generic and easy-to-decode perceptual state in an end-to-end RL framework. We show that they aid generalization, improve sample complexity, and lead to a higher final performance. In practice, this means that mid-level representations could be used to successfully train policies for tasks where domain randomization and learning-from-scratch failed.
  arXiv  Detail & Related papers  (2020-11-13T00:16:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.