Masked Diffusion Vision-Language Models for Temporal Action Localization

• URL: http://arxiv.org/abs/2605.29858v1
• Date: Thu, 28 May 2026 12:39:04 GMT
• Title: Masked Diffusion Vision-Language Models for Temporal Action Localization
• Abstract summary: Temporal action localization (TAL) requires recognizing the target event and localizing its start and end times precisely in untrimmed videos.<n>Recent vision-language formulations improve semantic reasoning and support language-conditioned outputs, but their autoregressive decoders still generate tokens from left to right.<n>We adapt vision-language models to TAL so that semantic tokens and boundary tokens remain editable throughout iterative denoising.
• Score: 10.018515245238216
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Temporal action localization (TAL) requires recognizing the target event and localizing its start and end times precisely in untrimmed videos. Recent vision-language formulations improve semantic reasoning and support language-conditioned outputs, but their autoregressive decoders still generate tokens from left to right, preventing later semantic evidence from revising earlier timestamp predictions. We adapt masked diffusion vision-language models (MDVLMs) to TAL so that semantic tokens and boundary tokens remain editable throughout iterative denoising with bidirectional attention, allowing temporal boundaries and semantic content to be refined jointly. Direct adaptation, however, creates two TAL-specific mismatches: standard masked diffusion training corrupts all positions uniformly at random, but the time tokens are more reliable when enough semantic context is available; and token-level cross-entropy does not reflect temporal IoU. To address these mismatches, we introduce a Planned Training Objective that uses boundary-aware masking and step-weighted reconstruction to rehearse the late recovery of time tokens, together with a Step-Level IoU Reward that provides overlap-aware supervision during denoising. A standard sequence-level cross-entropy term provides the base reconstruction signal. Experiments on ActivityNet-RTL, ActivityNet-1.3, and THUMOS-14 show that MDVLM-TAL improves both temporal reasoning and boundary localization over autoregressive vision-language baselines, with especially strong gains under stricter temporal IoU criteria.

Related papers

• An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation [55.291219073365546]
  We introduce textbfMulti-temporal Referring (MTRS), a new task that aims to segment language-described temporal changes from multi-temporal images.<n>MTRS extends conventional referring segmentation and change detection by jointly requiring temporal correspondence reasoning, language grounding, and pixel-level mask prediction.<n>We show that MTRefSeg-R1 achieves strong and often superior performance compared with existing LVLM baselines.
  arXiv  Detail & Related papers  (2026-05-31T04:01:10Z)
• Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models [7.964052580720558]
  We show that LDVLMs suffer from repetitive generation and degraded visual grounding.<n>We propose a training-free approach, introducing Mask Prior Suppression and Monotonic RoPE Scaling.<n>Our results show that these failures can be effectively addressed with a lightweight, plug-and-play strategy.
  arXiv  Detail & Related papers  (2026-05-14T08:11:32Z)
• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference [44.87788417755154]
  We propose WeDLM, a diffusion decoding framework built entirely on standard causal attention.<n>We show that WeDLM preserves the quality of strong AR backbones while delivering substantial speedups.
  arXiv  Detail & Related papers  (2025-12-28T01:25:48Z)
• 1 + 1 > 2: Detector-Empowered Video Large Language Model for Spatio-Temporal Grounding and Reasoning [53.28271278708241]
  We present a Detector-Empowered Video LLM, short for DEViL.<n> DEViL couples a Video LLM with an open-vocabulary detector (OVD)<n>Unlike tokens that merely serve as spatial prompts or segmentor switches, the RST functions as both a control signal and a replacement for the OVD's text embedding.
  arXiv  Detail & Related papers  (2025-12-07T06:11:15Z)
• Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding [30.223279362023337]
  Video Grounding (VTG) aims to localize temporal segments in long, untrimmed videos that align with a given natural language query.<n>Existing approaches commonly treat all text tokens uniformly during crossmodal attention, disregarding their distinct semantic roles.<n>We propose DualGround, a dual-branch architecture that explicitly separates global and local semantics.
  arXiv  Detail & Related papers  (2025-10-23T05:53:01Z)
• Temporal Grounding as a Learning Signal for Referring Video Object Segmentation [29.646697516547558]
  Referring Video Object (RVOS) aims to segment and track objects in videos based on natural language expressions, requiring precise alignment between visual content and textual queries.<n>Existing methods often suffer from semantic misalignment, largely due to indiscriminate frame sampling and supervision of all visible objects during training.<n>We introduce MeViS-M, a dataset built upon the challenging MeViS benchmark, where we manually annotate temporal spans when each object is referred to by the expression.
  arXiv  Detail & Related papers  (2025-08-16T07:34:43Z)
• From Values to Tokens: An LLM-Driven Framework for Context-aware Time Series Forecasting via Symbolic Discretization [21.8427780153806]
  Time series forecasting plays a vital role in supporting decision-making across a wide range of critical applications, including energy, healthcare, and finance.<n>We propose TokenCast, an LLM-driven framework that leverages language-based symbolic representations as a unified intermediary for context-aware time series forecasting.<n>Specifically, TokenCast employs a discrete tokenizer to transform continuous numerical sequences into temporal tokens, enabling structural alignment with language-based inputs.
  arXiv  Detail & Related papers  (2025-08-08T03:51:08Z)
• Collaborative Temporal Consistency Learning for Point-supervised Natural Language Video Localization [129.43937834515688]
  We propose a new COllaborative Temporal consistEncy Learning (COTEL) framework to strengthen the video-language alignment.<n>Specifically, we first design a frame- and a segment-level Temporal Consistency Learning (TCL) module that models semantic alignment across frame saliencies and sentence-moment pairs.
  arXiv  Detail & Related papers  (2025-03-22T05:04:12Z)
• Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding [112.3913646778859]
  We propose a simple yet effective video-language modeling framework, S-ViLM. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features. S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T22:45:07Z)
• Weakly Supervised Temporal Adjacent Network for Language Grounding [96.09453060585497]
  We introduce a novel weakly supervised temporal adjacent network (WSTAN) for temporal language grounding. WSTAN learns cross-modal semantic alignment by exploiting temporal adjacent network in a multiple instance learning (MIL) paradigm. An additional self-discriminating loss is devised on both the MIL branch and the complementary branch, aiming to enhance semantic discrimination by self-supervising.
  arXiv  Detail & Related papers  (2021-06-30T15:42:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.