Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning

• URL: http://arxiv.org/abs/2410.14038v3
• Date: Thu, 13 Feb 2025 19:38:48 GMT
• Title: Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
• Authors: Bryan L. M. de Oliveira, Murilo L. da Luz, Bruno Brandão, Luana G. B. Martins, Telma W. de L. Soares, Luckeciano C. Melo,
• Abstract summary: We introduce the Sliding Puzzles Gym (SPGym), a novel benchmark that reimagines the classic 8-tile puzzle with a visual observation space of images sourced from arbitrarily large datasets. SPGym provides precise control over representation complexity through visual diversity, allowing researchers to systematically scale the representation learning challenge. As we increase visual diversity by expanding the pool of possible images, all tested algorithms show significant performance degradation.
• Score: 3.8309622155866583
• License:
• Abstract: Learning effective visual representations enables agents to extract meaningful information from raw sensory inputs, which is essential for generalizing across different tasks. However, evaluating representation learning separately from policy learning remains a challenge with most reinforcement learning (RL) benchmarks. To address this gap, we introduce the Sliding Puzzles Gym (SPGym), a novel benchmark that reimagines the classic 8-tile puzzle with a visual observation space of images sourced from arbitrarily large datasets. SPGym provides precise control over representation complexity through visual diversity, allowing researchers to systematically scale the representation learning challenge while maintaining consistent environment dynamics. Despite the apparent simplicity of the task, our experiments with both model-free and model-based RL algorithms reveal fundamental limitations in current methods. As we increase visual diversity by expanding the pool of possible images, all tested algorithms show significant performance degradation, with even state-of-the-art methods struggling to generalize across different visual inputs while maintaining consistent puzzle-solving capabilities. These results highlight critical gaps in visual representation learning for RL and provide clear directions for improving robustness and generalization in decision-making systems.

Related papers

• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models [81.71651422951074]
  Chain-of-Spot (CoS) method is a novel approach that enhances feature extraction by focusing on key regions of interest. This technique allows LVLMs to access more detailed visual information without altering the original image resolution. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content.
  arXiv  Detail & Related papers  (2024-03-19T17:59:52Z)
• Improving Reinforcement Learning Efficiency with Auxiliary Tasks in Non-Visual Environments: A Comparison [0.0]
  This study compares common auxiliary tasks based on, to the best of our knowledge, the only decoupled representation learning method for low-dimensional non-visual observations. Our findings show that representation learning with auxiliary tasks only provides performance gains in sufficiently complex environments.
  arXiv  Detail & Related papers  (2023-10-06T13:22:26Z)
• VIBR: Learning View-Invariant Value Functions for Robust Visual Control [3.2307366446033945]
  VIBR (View-Invariant Bellman Residuals) is a method that combines multi-view training and invariant prediction to reduce out-of-distribution gap for RL based visuomotor control. We show that VIBR outperforms existing methods on complex visuo-motor control environment with high visual perturbation.
  arXiv  Detail & Related papers  (2023-06-14T14:37:34Z)
• Accelerating exploration and representation learning with offline pre-training [52.6912479800592]
  We show that exploration and representation learning can be improved by separately learning two different models from a single offline dataset. We show that learning a state representation using noise-contrastive estimation and a model of auxiliary reward can significantly improve the sample efficiency on the challenging NetHack benchmark.
  arXiv  Detail & Related papers  (2023-03-31T18:03:30Z)
• Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems [61.11799513362704]
  We propose learning an additional screening mechanism to identify discriminative clues commonly seen across instances and classes. We show that a common rationale detector can be learned by simply exploiting the GradCAM induced from the SSL objective.
  arXiv  Detail & Related papers  (2023-03-03T02:07:40Z)
• Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search [63.3745291252038]
  We propose DiffSES, a novel symbolic learning approach that discovers discrete symbolic policies. By using object-level abstractions instead of raw pixel-level inputs, DiffSES is able to leverage the simplicity and scalability advantages of symbolic expressions. Our experiments demonstrate that DiffSES is able to generate symbolic policies that are simpler and more scalable than state-of-the-art symbolic RL methods.
  arXiv  Detail & Related papers  (2022-12-30T17:50:54Z)
• Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem [60.0878532426877]
  We propose a novel collaborative learning scheme from the viewpoint of visual perturbation calibration. Specifically, we devise a visual controller to construct two sorts of curated images with different perturbation extents. The experimental results on two diagnostic VQA-CP benchmark datasets evidently demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2022-07-24T23:50:52Z)
• Task-Induced Representation Learning [14.095897879222672]
  We evaluate the effectiveness of representation learning approaches for decision making in visually complex environments. We find that representation learning generally improves sample efficiency on unseen tasks even in visually complex scenes.
  arXiv  Detail & Related papers  (2022-04-25T17:57:10Z)
• X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation [71.51719469058666]
  We propose a representation learning framework called X-Learner. X-Learner learns the universal feature of multiple vision tasks supervised by various sources. X-Learner achieves strong performance on different tasks without extra annotations, modalities and computational costs.
  arXiv  Detail & Related papers  (2022-03-16T17:23:26Z)
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels [10.727930028878516]
  We extend DM Control with three kinds of visual distractions to produce a new challenging benchmark for vision-based control. Our experiments show that current RL methods for vision-based control perform poorly under distractions. We also find that combinations of multiple distraction types are more difficult than a mere combination of their individual effects.
  arXiv  Detail & Related papers  (2021-01-07T19:03:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.