What Do Latent Action Models Actually Learn?

• URL: http://arxiv.org/abs/2506.15691v1
• Date: Tue, 27 May 2025 02:45:30 GMT
• Title: What Do Latent Action Models Actually Learn?
• Abstract summary: Latent action models (LAMs) aim to learn action-relevant changes from unlabeled videos by compressing changes between frames as latents.<n>This paper studies this issue analytically, presenting a linear model that encapsulates the essence of LAM learning, while being tractable.
• Score: 26.1770533810057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Latent action models (LAMs) aim to learn action-relevant changes from unlabeled videos by compressing changes between frames as latents. However, differences between video frames can be caused by controllable changes as well as exogenous noise, leading to an important concern -- do latents capture the changes caused by actions or irrelevant noise? This paper studies this issue analytically, presenting a linear model that encapsulates the essence of LAM learning, while being tractable.This provides several insights, including connections between LAM and principal component analysis (PCA), desiderata of the data-generating policy, and justification of strategies to encourage learning controllable changes using data augmentation, data cleaning, and auxiliary action-prediction. We also provide illustrative results based on numerical simulation, shedding light on the specific structure of observations, actions, and noise in data that influence LAM learning.

Related papers

• Segment to Focus: Guiding Latent Action Models in the Presence of Distractors [0.8411385346896413]
  Latent Action Models learn to extract action-relevant representations solely from raw observations.<n>LAMs face a critical challenge in disentangling action-relevant features from action-correlated noise.<n>We introduce MaskLAM -- a lightweight modification to LAM training to mitigate this issue by incorporating visual agent segmentation.
  arXiv  Detail & Related papers  (2026-02-02T16:03:19Z)
• Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering [22.666436755894328]
  Large language models (LLMs) can be controlled at inference time through prompts (in-context learning) and internal activations (activation steering)<n>This work offers a unified account of prompt-based and activation-based control of LLM behavior, and a methodology for empirically predicting the effects of these interventions.
  arXiv  Detail & Related papers  (2025-11-01T16:46:03Z)
• Dynamic Evaluation for Oversensitivity in LLMs [68.27609301865174]
  Oversensitivity occurs when language models defensively reject prompts that are actually benign.<n>This behavior not only disrupts user interactions but also obscures the boundary between harmful and harmless content.<n>Existing benchmarks rely on static datasets that degrade overtime as models evolve.
  arXiv  Detail & Related papers  (2025-10-21T18:33:47Z)
• Latent Variable Modeling for Robust Causal Effect Estimation [10.245287265072927]
  We propose a new framework that integrates latent variable modeling into the double machine learning paradigm.<n>We demonstrate the robustness and effectiveness of our method through extensive experiments on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2025-08-27T20:31:03Z)
• Mitigating Spurious Correlations with Causal Logit Perturbation [22.281052412112263]
  This study introduces a novel Causal Logit Perturbation (CLP) framework to train classifiers with generated causal logit perturbations for individual samples.<n>The framework is optimized by an online meta-learning-based learning algorithm and leverages human causal knowledge by augmenting metadata in both counterfactual and factual manners.
  arXiv  Detail & Related papers  (2025-05-21T08:21:02Z)
• Model Hemorrhage and the Robustness Limits of Large Language Models [119.46442117681147]
  Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment.<n>We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes.
  arXiv  Detail & Related papers  (2025-03-31T10:16:03Z)
• PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policies [7.376615925443845]
  We introduce PrivilegedDreamer, a model-based reinforcement learning framework that extends the existing model-based approach by incorporating an explicit parameter estimation module.<n>Our empirical analysis on five diverse HIP-MDP tasks demonstrates that PrivilegedDreamer outperforms state-of-the-art model-based, model-free, and do- main adaptation learning algorithms.
  arXiv  Detail & Related papers  (2025-02-17T02:46:02Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning [80.44084021062105]
  We propose a novel latent partial causal model for multimodal data, featuring two latent coupled variables, connected by an undirected edge, to represent the transfer of knowledge across modalities.<n>Under specific statistical assumptions, we establish an identifiability result, demonstrating that representations learned by multimodal contrastive learning correspond to the latent coupled variables up to a trivial transformation.<n>Experiments on a pre-trained CLIP model embodies disentangled representations, enabling few-shot learning and improving domain generalization across diverse real-world datasets.
  arXiv  Detail & Related papers  (2024-02-09T07:18:06Z)
• Unraveling the Temporal Dynamics of the Unet in Diffusion Models [33.326244121918634]
  Diffusion models introduce Gaussian noise into training data and reconstruct the original data iteratively. Central to this iterative process is a single Unet, adapting across time steps to facilitate generation. Recent work revealed the presence of composition and denoising phases in this generation process.
  arXiv  Detail & Related papers  (2023-12-17T04:40:33Z)
• Identifiable Latent Polynomial Causal Models Through the Lens of Change [82.14087963690561]
  Causal representation learning aims to unveil latent high-level causal representations from observed low-level data.<n>One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability.
  arXiv  Detail & Related papers  (2023-10-24T07:46:10Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• Treatment Learning Causal Transformer for Noisy Image Classification [62.639851972495094]
  In this work, we incorporate this binary information of "existence of noise" as treatment into image classification tasks to improve prediction accuracy. Motivated from causal variational inference, we propose a transformer-based architecture, that uses a latent generative model to estimate robust feature representations for noise image classification. We also create new noisy image datasets incorporating a wide range of noise factors for performance benchmarking.
  arXiv  Detail & Related papers  (2022-03-29T13:07:53Z)
• Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updates [26.527311287924995]
  influence tuning can help deconfounding the model from spurious patterns in data. We show that in a controlled setup, influence tuning can help deconfounding the model from spurious patterns in data.
  arXiv  Detail & Related papers  (2021-10-07T06:59:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.