Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data

• URL: http://arxiv.org/abs/2508.12356v1
• Date: Sun, 17 Aug 2025 13:01:15 GMT
• Title: Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data
• Authors: Ahmet H. Güzel, Ilija Bogunovic, Jack Parker-Holder,
• Abstract summary: policies trained on offline data often struggle to generalise due to limited exposure to diverse states.<n>This makes it challenging to leverage vision-based offline data in training robust agents that can generalize to unseen environments.<n>We propose a two-step process, first augmenting the originally collected offline data to improve zero-shot generalization by introducing diversity, then using a diffusion model to generate additional data in latent space.
• Score: 22.840912154067325
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Offline reinforcement learning (RL) offers a promising framework for training agents using pre-collected datasets without the need for further environment interaction. However, policies trained on offline data often struggle to generalise due to limited exposure to diverse states. The complexity of visual data introduces additional challenges such as noise, distractions, and spurious correlations, which can misguide the policy and increase the risk of overfitting if the training data is not sufficiently diverse. Indeed, this makes it challenging to leverage vision-based offline data in training robust agents that can generalize to unseen environments. To solve this problem, we propose a simple approach generating additional synthetic training data. We propose a two-step process, first augmenting the originally collected offline data to improve zero-shot generalization by introducing diversity, then using a diffusion model to generate additional data in latent space. We test our method across both continuous action spaces (Visual D4RL) and discrete action spaces (Procgen), demonstrating that it significantly improves generalization without requiring any algorithmic changes to existing model-free offline RL methods. We show that our method not only increases the diversity of the training data but also significantly reduces the generalization gap at test time while maintaining computational efficiency. We believe this approach could fuel additional progress in generating synthetic data to train more general agents in the future.

Related papers

• When Dynamic Data Selection Meets Data Augmentation [10.217776379089093]
  We propose a novel online data training framework that unifies dynamic data selection and augmentation.<n>Our method estimates each sample's joint distribution of local density and multimodal semantic consistency, allowing for the targeted selection of augmentation-suitable samples.<n>Our approach enhances noise resistance and improves model robustness, reinforcing its practical utility in real-world scenarios.
  arXiv  Detail & Related papers  (2025-05-02T11:38:48Z)
• Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation [36.9134885948595]
  We introduce Model-based Offline Reinforcement learning with AdversariaL data augmentation.<n>In MORAL, we replace the fixed horizon rollout by employing adversaria data augmentation to execute alternating sampling with ensemble models.<n>Experiments on D4RL benchmark demonstrate that MORAL outperforms other model-based offline RL methods in terms of policy learning and sample efficiency.
  arXiv  Detail & Related papers  (2025-03-26T07:24:34Z)
• Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration [54.8229698058649]
  Unlabeled offline trajectory data can be leveraged to learn efficient exploration strategies.<n>Our method SUPE consistently outperforms prior strategies across a suite of 42 long-horizon, sparse-reward tasks.
  arXiv  Detail & Related papers  (2024-10-23T17:58:45Z)
• Zero-Shot Generalization of Vision-Based RL Without Data Augmentation [11.820012065797917]
  Generalizing vision-based reinforcement learning (RL) agents to novel environments remains a difficult and open challenge.<n>We propose a model, Associative Latent DisentAnglement (ALDA), that builds on standard off-policy RL towards zero-shot generalization.
  arXiv  Detail & Related papers  (2024-10-09T21:14:09Z)
• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning [99.33607114541861]
  We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:27:00Z)
• Are Synthetic Time-series Data Really not as Good as Real Data? [29.852306720544224]
  Time-series data presents limitations stemming from data quality issues, bias and vulnerabilities, and generalization problem. We introduce InfoBoost -- a highly versatile cross-domain data synthesizing framework with time series representation learning capability. We have developed a method based on synthetic data that enables model training without the need for real data, surpassing the performance of models trained with real data.
  arXiv  Detail & Related papers  (2024-02-01T13:59:04Z)
• Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation [59.899714450049494]
  offline pre-training can produce sub-optimal policies and lead to degraded online reinforcement learning performance. We propose a model-based data augmentation strategy to maximize the benefits of offline reinforcement learning pre-training and reduce the scale of data needed to be effective.
  arXiv  Detail & Related papers  (2023-12-15T14:49:41Z)
• Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling [11.751910133386254]
  Recent advances in batch (offline) reinforcement learning have shown promising results in learning from available offline data. We propose a novel approach that uses uncertainty estimation to trigger the injection of human demonstration data. Our experiments show that this approach is more sample efficient when compared to a naive way of combining expert data with data collected from a sub-optimal agent.
  arXiv  Detail & Related papers  (2022-12-16T01:41:59Z)
• Parallel Augmentation and Dual Enhancement for Occluded Person Re-identification [70.96277129480478]
  Occluded person re-identification (Re-ID) has attracted lots of attention in the past decades. Recent approaches concentrate on improving performance on occluded data. We propose a simple yet effective method with Parallel Augmentation and Dual Enhancement (PADE) Experimental results on three widely used occluded datasets and two non-occluded datasets validate the effectiveness of our method.
  arXiv  Detail & Related papers  (2022-10-11T13:29:38Z)
• Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning [147.61075994259807]
  We propose Exploratory data for Offline RL (ExORL), a data-centric approach to offline RL. ExORL first generates data with unsupervised reward-free exploration, then relabels this data with a downstream reward before training a policy with offline RL. We find that exploratory data allows vanilla off-policy RL algorithms, without any offline-specific modifications, to outperform or match state-of-the-art offline RL algorithms on downstream tasks.
  arXiv  Detail & Related papers  (2022-01-31T18:39:27Z)
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL [82.93243616342275]
  We introduce Offline Model-based RL with Adaptive Behavioral Priors (MABE) MABE is based on the finding that dynamics models, which support within-domain generalization, and behavioral priors, which support cross-domain generalization, are complementary. In experiments that require cross-domain generalization, we find that MABE outperforms prior methods.
  arXiv  Detail & Related papers  (2021-06-16T20:48:49Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.