EgoVIS@CVPR: What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning

• URL: http://arxiv.org/abs/2506.00101v1
• Date: Fri, 30 May 2025 13:39:29 GMT
• Title: EgoVIS@CVPR: What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning
• Authors: Chi-Hsi Kung, Frangil Ramirez, Juhyung Ha, Yi-Ting Chen, David Crandall, Yi-Hsuan Tsai,
• Abstract summary: We study procedure-aware video representation learning by incorporating state-change descriptions.<n>We generate state-change counterfactuals that simulate hypothesized failure outcomes.<n>Our results demonstrate the effectiveness of the proposed state-change descriptions and their counterfactuals.
• Score: 22.00652926645987
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding a procedural activity requires modeling both how action steps transform the scene, and how evolving scene transformations can influence the sequence of action steps, even those that are accidental or erroneous. Yet, existing work on procedure-aware video representations fails to explicitly learned the state changes (scene transformations). In this work, we study procedure-aware video representation learning by incorporating state-change descriptions generated by LLMs as supervision signals for video encoders. Moreover, we generate state-change counterfactuals that simulate hypothesized failure outcomes, allowing models to learn by imagining the unseen ``What if'' scenarios. This counterfactual reasoning facilitates the model's ability to understand the cause and effect of each step in an activity. To verify the procedure awareness of our model, we conduct extensive experiments on procedure-aware tasks, including temporal action segmentation, error detection, and more. Our results demonstrate the effectiveness of the proposed state-change descriptions and their counterfactuals, and achieve significant improvements on multiple tasks.

Related papers

• What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning [22.00652926645987]
  We study procedure-aware video representation learning by incorporating state-change descriptions.<n>We generate state-change counterfactuals that simulate hypothesized failure outcomes.<n>Our results demonstrate the effectiveness of the proposed state-change descriptions and their counterfactuals.
  arXiv  Detail & Related papers  (2025-03-27T00:03:55Z)
• Learning Actionable World Models for Industrial Process Control [5.870452455598225]
  An effective AI system must learn about the behavior of the complex system from very limited training data.<n>We propose a novel methodology that disentangles process parameters in the learned latent representation.<n>This makes changes in representations predictable from changes in inputs and vice versa, facilitating interpretability of key factors responsible for process variations.
  arXiv  Detail & Related papers  (2025-03-03T11:05:44Z)
• SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos [54.01116513202433]
  We study the problem of procedure planning in instructional videos, which aims to make a goal-oriented sequence of action steps given partial visual state observations. Recent works succeeded in sequence modeling of steps with only sequence-level annotations accessible during training, which overlooked the roles of states in the procedures. We aim to establish a more structured state space by investigating the causal relations between steps and states in procedures.
  arXiv  Detail & Related papers  (2024-03-03T19:53:06Z)
• Masked Diffusion with Task-awareness for Procedure Planning in Instructional Videos [16.93979476655776]
  A key challenge with procedure planning in instructional videos is how to handle a large decision space consisting of a multitude of action types. We introduce a simple yet effective enhancement - a masked diffusion model. We learn a joint visual-text embedding, where a text embedding is generated by prompting a pre-trained vision-language model to focus on human actions.
  arXiv  Detail & Related papers  (2023-09-14T03:25:37Z)
• STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videos [40.82053186029603]
  We decompose the problem into two steps: representation learning and key steps extraction. We propose a training objective, Bootstrapped Multi-Cue Contrastive (BMC2) loss to learn discriminative representations for various steps without any labels. We show significant improvements over prior works for the task of key step localization and phase classification.
  arXiv  Detail & Related papers  (2023-01-02T18:32:45Z)
• Multi-Task Learning of Object State Changes from Uncurated Videos [55.60442251060871]
  We learn to temporally localize object state changes by observing people interacting with objects in long uncurated web videos. We show that our multi-task model achieves a relative improvement of 40% over the prior single-task methods. We also test our method on long egocentric videos of the EPIC-KITCHENS and the Ego4D datasets in a zero-shot setup.
  arXiv  Detail & Related papers  (2022-11-24T09:42:46Z)
• Leveraging Self-Supervised Training for Unintentional Action Recognition [82.19777933440143]
  We seek to identify the points in videos where the actions transition from intentional to unintentional. We propose a multi-stage framework that exploits inherent biases such as motion speed, motion direction, and order to recognize unintentional actions.
  arXiv  Detail & Related papers  (2022-09-23T21:36:36Z)
• P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision [31.73732506824829]
  We study the problem of procedure planning in instructional videos. Here, an agent must produce a plausible sequence of actions that can transform the environment from a given start to a desired goal state. We propose a weakly supervised approach by learning from natural language instructions.
  arXiv  Detail & Related papers  (2022-05-04T19:37:32Z)
• Transformers in Action:Weakly Supervised Action Segmentation [81.18941007536468]
  We show how to apply transformers to improve action alignment accuracy over the equivalent RNN-based models. We also propose a supplemental transcript embedding approach to select transcripts more quickly at inference-time. We evaluate our proposed methods across the benchmark datasets to better understand the applicability of transformers.
  arXiv  Detail & Related papers  (2022-01-14T21:15:58Z)
• Procedure Planning in Instructional Videosvia Contextual Modeling and Model-based Policy Learning [114.1830997893756]
  This work focuses on learning a model to plan goal-directed actions in real-life videos. We propose novel algorithms to model human behaviors through Bayesian Inference and model-based Imitation Learning.
  arXiv  Detail & Related papers  (2021-10-05T01:06:53Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.