InstrAct: Towards Action-Centric Understanding in Instructional Videos

• URL: http://arxiv.org/abs/2604.08762v1
• Date: Thu, 09 Apr 2026 20:51:13 GMT
• Title: InstrAct: Towards Action-Centric Understanding in Instructional Videos
• Abstract summary: InstrAction is a pretraining framework for instructional videos' action-centric representations.<n>We first introduce a data-driven strategy, which filters noisy captions and generates action-centric hard negatives.<n>At the visual feature level, an Action Perceiver extracts motion-relevant tokens from redundant video encodings.
• Score: 12.356484522873577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding instructional videos requires recognizing fine-grained actions and modeling their temporal relations, which remains challenging for current Video Foundation Models (VFMs). This difficulty stems from noisy web supervision and a pervasive "static bias", where models rely on objects rather than motion cues. To address this, we propose InstrAction, a pretraining framework for instructional videos' action-centric representations. We first introduce a data-driven strategy, which filters noisy captions and generates action-centric hard negatives to disentangle actions from objects during contrastive learning. At the visual feature level, an Action Perceiver extracts motion-relevant tokens from redundant video encodings. Beyond contrastive learning, we introduce two auxiliary objectives: Dynamic Time Warping alignment (DTW-Align) for modeling sequential temporal structure, and Masked Action Modeling (MAM) for strengthening cross-modal grounding. Finally, we introduce the InstrAct Bench to evaluate action-centric understanding, where our method consistently outperforms state-of-the-art VFMs on semantic reasoning, procedural logic, and fine-grained retrieval tasks.

Related papers

• Local Motion Matters: A Deconstruct-Recompose Paradigm for Reinforcement Learning Pre-training from Videos [30.202149567220356]
  Existing methods typically treat the agent as an indivisible entity, modeling motion patterns globally.<n>We propose a novel Deconstruct-compose Paradigm for learning transferable local motion representations.<n>Our method effectively transfers to diverse robotic control and manipulation tasks, significantly improving sample efficiency and performance.
  arXiv  Detail & Related papers  (2026-07-01T11:36:27Z)
• OR-Action: Multi-Role Video Understanding with Fine-Grained Actions [37.96856087875045]
  Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance.<n>Current scene graph prediction methods struggle to model temporal structure.<n>We introduce a vision-only temporal model that outperforms graph-based methods.
  arXiv  Detail & Related papers  (2026-06-11T13:24:40Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• TrackMAE: Video Representation Learning via Track Mask and Predict [53.79942817343784]
  Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm.<n>We propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal.<n>We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms state-of-the-art video self-supervised learning baselines.
  arXiv  Detail & Related papers  (2026-03-28T13:35:23Z)
• Zero-Shot Open-Vocabulary Human Motion Grounding with Test-Time Training [39.7658823121591]
  ZOMG is a framework that segments motion sequences into semantically meaningful sub-actions without requiring any annotations or fine-tuning.<n>ZOMG integrates (1) language semantic partition, which leverages large language models to decompose instructions into ordered sub-action units, and (2) soft masking optimization.<n>Experiments on three motion-language datasets demonstrate state-of-the-art effectiveness and efficiency of motion grounding performance, outperforming prior methods by +8.7% mAP on HumanML3D benchmark.
  arXiv  Detail & Related papers  (2025-11-19T12:11:36Z)
• SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation [56.90807453045657]
  SynMotion is a motion-customized video generation model that jointly leverages semantic guidance and visual adaptation.<n>At the semantic level, we introduce the dual-em semantic comprehension mechanism which disentangles subject and motion representations.<n>At the visual level, we integrate efficient motion adapters into a pre-trained video generation model to enhance motion fidelity and temporal coherence.
  arXiv  Detail & Related papers  (2025-06-30T10:09:32Z)
• CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning [47.195002937893115]
  CoMo aims to learn more informative continuous motion representations from diverse, internet-scale videos.<n>We introduce two new metrics for more robustly and affordably evaluating motion and guiding motion learning methods.<n>CoMo exhibits strong zero-shot generalization, enabling it to generate continuous pseudo actions for previously unseen video domains.
  arXiv  Detail & Related papers  (2025-05-22T17:58:27Z)
• Capturing Rich Behavior Representations: A Dynamic Action Semantic-Aware Graph Transformer for Video Captioning [13.411096520754507]
  Existing video captioning methods merely provide shallow or simplistic representations of object behaviors.<n>We propose a dynamic action semantic-aware graph transformer to comprehensively capture the essence of object behavior.
  arXiv  Detail & Related papers  (2025-02-19T14:16:47Z)
• ActPrompt: In-Domain Feature Adaptation via Action Cues for Video Temporal Grounding [40.60371529725805]
  We propose an efficient preliminary in-domain fine-tuning paradigm for feature adaptation. We introduce Action-Cue-Injected Temporal Prompt Learning (ActPrompt), which injects action cues into the image encoder of VLM for better discovering action-sensitive patterns.
  arXiv  Detail & Related papers  (2024-08-13T04:18:32Z)
• Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding [112.3913646778859]
  We propose a simple yet effective video-language modeling framework, S-ViLM. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features. S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T22:45:07Z)
• Exploring Motion and Appearance Information for Temporal Sentence Grounding [52.01687915910648]
  We propose a Motion-Appearance Reasoning Network (MARN) to solve temporal sentence grounding. We develop separate motion and appearance branches to learn motion-guided and appearance-guided object relations. Our proposed MARN significantly outperforms previous state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-01-03T02:44:18Z)
• Background-Click Supervision for Temporal Action Localization [82.4203995101082]
  Weakly supervised temporal action localization aims at learning the instance-level action pattern from the video-level labels, where a significant challenge is action-context confusion. One recent work builds an action-click supervision framework. It requires similar annotation costs but can steadily improve the localization performance when compared to the conventional weakly supervised methods. In this paper, by revealing that the performance bottleneck of the existing approaches mainly comes from the background errors, we find that a stronger action localizer can be trained with labels on the background video frames rather than those on the action frames.
  arXiv  Detail & Related papers  (2021-11-24T12:02:52Z)
• Towards Tokenized Human Dynamics Representation [41.75534387530019]
  We study how to segment and cluster videos into recurring temporal patterns in a self-supervised way. We evaluate the frame-wise representation learning step by Kendall's Tau and the lexicon building step by normalized mutual information and language entropy. On the AIST++ and PKU-MMD datasets, actons bring significant performance improvements compared to several baselines.
  arXiv  Detail & Related papers  (2021-11-22T18:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.