Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies

• URL: http://arxiv.org/abs/2510.05692v1
• Date: Tue, 07 Oct 2025 08:49:31 GMT
• Title: Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies
• Authors: Yuhang Zhang, Jiaping Xiao, Chao Yan, Mir Feroskhan,
• Abstract summary: A prevailing approach for learning visuomotor policies is to employ reinforcement learning to map high-dimensional visual observations directly to action commands.<n>We propose Oracle-Guided Masked Contrastive Reinforcement Learning (OMC-RL), a novel framework designed to improve the sample efficiency and performance of visuomotor policy learning.
• Score: 9.663452274930643
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A prevailing approach for learning visuomotor policies is to employ reinforcement learning to map high-dimensional visual observations directly to action commands. However, the combination of high-dimensional visual inputs and agile maneuver outputs leads to long-standing challenges, including low sample efficiency and significant sim-to-real gaps. To address these issues, we propose Oracle-Guided Masked Contrastive Reinforcement Learning (OMC-RL), a novel framework designed to improve the sample efficiency and asymptotic performance of visuomotor policy learning. OMC-RL explicitly decouples the learning process into two stages: an upstream representation learning stage and a downstream policy learning stage. In the upstream stage, a masked Transformer module is trained with temporal modeling and contrastive learning to extract temporally-aware and task-relevant representations from sequential visual inputs. After training, the learned encoder is frozen and used to extract visual representations from consecutive frames, while the Transformer module is discarded. In the downstream stage, an oracle teacher policy with privileged access to global state information supervises the agent during early training to provide informative guidance and accelerate early policy learning. This guidance is gradually reduced to allow independent exploration as training progresses. Extensive experiments in simulated and real-world environments demonstrate that OMC-RL achieves superior sample efficiency and asymptotic policy performance, while also improving generalization across diverse and perceptually complex scenarios.

Related papers

• Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning [11.901989132359676]
  We introduce Enhanced Semantic Motion Representations (Semore), a new VLM-based framework for visual reinforcement learning (RL)<n>Semore simultaneously extract semantic and motion representations through a dual-path backbone from the RGB flows.<n>Our method exhibits efficient and adaptive ability compared to state-of-art methods.
  arXiv  Detail & Related papers  (2025-12-04T16:54:41Z)
• Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent [72.1517476116743]
  Recent MLLMs have shown emerging visual understanding and reasoning abilities after being pre-trained on large-scale multimodal datasets.<n>Existing approaches, such as direct fine-tuning and continual learning methods, fail to explicitly address this issue.<n>We introduce a novel perspective leveraging effective rank to quantify the degradation of visual representation forgetting.<n>We propose a modality-decoupled gradient descent (MDGD) method that regulates gradient updates to maintain the effective rank of visual representations.
  arXiv  Detail & Related papers  (2025-02-17T12:26:34Z)
• ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy [39.06557194970261]
  ImitDiff is a diffusion-based imitation learning policy guided by fine-grained semantics.<n>Our method transforms high-level instructions into pixel-level visual semantic masks.<n>ImitDiff exhibits strong generalization in zero-shot settings involving novel objects and visual distractions.
  arXiv  Detail & Related papers  (2025-02-11T14:03:57Z)
• Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learning [12.277005054008017]
  In visual Reinforcement Learning (RL), upstream representation learning largely determines the effect of downstream policy learning. We try to improve auxiliary representation learning for RL by enriching auxiliary training data. We propose a training-free method to synthesize observations that may contain future information. The remaining synthetic observations and real observations then serve as the auxiliary data to achieve a clustering-based temporal association task.
  arXiv  Detail & Related papers  (2024-05-20T02:43:04Z)
• Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight [20.92646531472541]
  We propose a novel approach that combines the performance of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL) Our framework contains three phases teacher policy using RL with privileged state information distilling it into a student policy via IL, and adaptive fine-tuning via RL. Tests show our approach can not only learn in scenarios where RL from scratch fails but also outperforms existing IL methods in both robustness and performance.
  arXiv  Detail & Related papers  (2024-03-18T19:25:57Z)
• TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning [73.53576440536682]
  We introduce TACO: Temporal Action-driven Contrastive Learning, a powerful temporal contrastive learning approach. TACO simultaneously learns a state and an action representation by optimizing the mutual information between representations of current states. For online RL, TACO achieves 40% performance boost after one million environment interaction steps.
  arXiv  Detail & Related papers  (2023-06-22T22:21:53Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learning [128.19212716007794]
  We propose an effective framework called textbfMulti-textbfAgent textbfMasked textbfAttentive textbfContrastive textbfLearning (MA2CL) MA2CL encourages learning representation to be both temporal and agent-level predictive by reconstructing the masked agent observation in latent space. Our method significantly improves the performance and sample efficiency of different MARL algorithms and outperforms other methods in various vision-based and state-based scenarios.
  arXiv  Detail & Related papers  (2023-06-03T05:32:19Z)
• Continual Visual Reinforcement Learning with A Life-Long World Model [55.05017177980985]
  We present a new continual learning approach for visual dynamics modeling.<n>We first introduce the life-long world model, which learns task-specific latent dynamics.<n>Then, we address the value estimation challenge for previous tasks with the exploratory-conservative behavior learning approach.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Reinforcement Learning with Prototypical Representations [114.35801511501639]
  Proto-RL is a self-supervised framework that ties representation learning with exploration through prototypical representations. These prototypes simultaneously serve as a summarization of the exploratory experience of an agent as well as a basis for representing observations. This enables state-of-the-art downstream policy learning on a set of difficult continuous control tasks.
  arXiv  Detail & Related papers  (2021-02-22T18:56:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.