Segment to Focus: Guiding Latent Action Models in the Presence of Distractors

• URL: http://arxiv.org/abs/2602.02259v1
• Date: Mon, 02 Feb 2026 16:03:19 GMT
• Title: Segment to Focus: Guiding Latent Action Models in the Presence of Distractors
• Abstract summary: Latent Action Models learn to extract action-relevant representations solely from raw observations.<n>LAMs face a critical challenge in disentangling action-relevant features from action-correlated noise.<n>We introduce MaskLAM -- a lightweight modification to LAM training to mitigate this issue by incorporating visual agent segmentation.
• Score: 0.8411385346896413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Latent Action Models (LAMs) learn to extract action-relevant representations solely from raw observations, enabling reinforcement learning from unlabelled videos and significantly scaling available training data. However, LAMs face a critical challenge in disentangling action-relevant features from action-correlated noise (e.g., background motion). Failing to filter these distractors causes LAMs to capture spurious correlations and build sub-optimal latent action spaces. In this paper, we introduce MaskLAM -- a lightweight modification to LAM training to mitigate this issue by incorporating visual agent segmentation. MaskLAM utilises segmentation masks from pretrained foundation models to weight the LAM reconstruction loss, thereby prioritising salient information over background elements while requiring no architectural modifications. We demonstrate the effectiveness of our method on continuous-control MuJoCo tasks, modified with action-correlated background noise. Our approach yields up to a 4x increase in accrued rewards compared to standard baselines and a 3x improvement in the latent action quality, as evidenced by linear probe evaluation.

Related papers

• Causally Debiased Latent Action Model for Embodied Action Conditioned World Models [47.281442841934876]
  Action-conditioned world models (ACWMs) aim to simulate future observations conditioned on embodied actions.<n>Existing LAMs are typically trained with reconstruction-only objectives.<n>We propose CD-LAM, a causally debiased framework for LAM-based ACWMs.
  arXiv  Detail & Related papers  (2026-07-10T08:20:27Z)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation [51.743225614196774]
  Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning.<n>They remain vulnerable to hallucination, where generated content deviates from visual evidence.<n>Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding.<n>We propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs.
  arXiv  Detail & Related papers  (2026-02-27T14:18:51Z)
• Dream-SLAM: Dreaming the Unseen for Active SLAM in Dynamic Environments [62.70468717776612]
  We propose a novel monocular active SLAM method, Dream-SLAM.<n>It is based on dreaming cross-spatio-temporal images and semantically plausible structures of partially observed dynamic environments.<n>Experiments on both public and self-collected datasets demonstrate that Dream-SLAM outperforms state-of-the-art methods in localization accuracy, mapping quality, and exploration efficiency.
  arXiv  Detail & Related papers  (2026-02-25T14:48:49Z)
• Vision-Language Models Unlock Task-Centric Latent Actions [75.53481518882275]
  We propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations.<n>We show that simply asking VLMs to ignore distractors can substantially improve latent action quality, yielding up to a six-fold increase in downstream success rates on Distracting MetaWorld.
  arXiv  Detail & Related papers  (2026-01-30T08:38:59Z)
• LVD-GS: Gaussian Splatting SLAM for Dynamic Scenes via Hierarchical Explicit-Implicit Representation Collaboration Rendering [21.615484471658842]
  3D Gaussian Splatting SLAM has emerged as a widely used technique for high-fidelity mapping in spatial intelligence.<n>Existing methods often rely on a single representation scheme, which limits their performance in large-scale dynamic outdoor scenes.<n>We propose textbfLVD-GS, a novel LiDAR-Visual 3D Gaussian Splatting SLAM system.
  arXiv  Detail & Related papers  (2025-10-26T13:16:39Z)
• RefAM: Attention Magnets for Zero-Shot Referral Segmentation [103.98022860792504]
  We introduce a new method that exploits features, attention scores, from diffusion transformers for downstream tasks.<n>Key insight is that stop words act as attention magnets.<n>We propose an attention redistribution strategy, where appended stop words partition background activations into smaller clusters.
  arXiv  Detail & Related papers  (2025-09-26T17:59:57Z)
• What Do Latent Action Models Actually Learn? [33.75212638602698]
  Latent action models (LAMs) aim to learn action-relevant changes from unlabeled videos by compressing changes between frames as latents.<n>This paper studies this issue analytically, presenting a linear model that encapsulates the essence of LAM learning, while being tractable.
  arXiv  Detail & Related papers  (2025-05-27T02:45:30Z)
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation [66.36912000442608]
  NoisyRollout is a simple yet effective data augmentation method.<n>It mixes training trajectories from both clean and moderately distorted images.<n>It achieves state-of-the-art performance among open-source RL-tuned models.
  arXiv  Detail & Related papers  (2025-04-17T16:10:13Z)
• Object-Centric Latent Action Learning [70.3173534658611]
  We propose a novel object-centric latent action learning framework that centers on objects rather than pixels.<n>We leverage self-supervised object-centric pretraining to disentangle action-related and distracting dynamics.<n>Our results show that object-centric pretraining mitigates the negative effects of distractors by 50%.
  arXiv  Detail & Related papers  (2025-02-13T11:27:05Z)
• Latent Action Learning Requires Supervision in the Presence of Distractors [40.33684677920241]
  We show that real-world videos contain action-correlated distractors that may hinder latent action learning.<n>We propose LAOM, a simple LAPO modification that improves the quality of latent actions by 8x.<n>We show that providing supervision with ground-truth actions, as few as 2.5% of the full dataset, during latent action learning improves downstream performance by 4.2x on average.
  arXiv  Detail & Related papers  (2025-02-01T09:35:51Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.