MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Grounding

• URL: http://arxiv.org/abs/2605.03398v1
• Date: Tue, 05 May 2026 06:20:33 GMT
• Title: MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Grounding
• Abstract summary: Video Temporal Grounding (VTG) faces a cross-modal semantic gap that often leads to background being incorrectly aligned with the query.<n>We propose MLLM-Assisted Semantic-Relational Consistent Alignment (MASRA), a training-time MLLM-based optimization framework for VTG.
• Score: 21.833707346184386
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Temporal Grounding (VTG) faces a cross-modal semantic gap that often leads to background features being incorrectly aligned with the query, while directly matching the query to moments results in insufficient discriminability and consistency of temporal semantics. To address this issue, we propose MLLM-Assisted Semantic-Relational Consistent Alignment (MASRA), a training-time MLLM-based optimization framework for VTG. MASRA leverages an MLLM during training to produce two forms of textual priors, namely event-level descriptions with temporal spans and clip-level captions, and instantiates two MLLM-assisted alignments. Event Semantic Temporal Alignment (ESTA) aligns temporal context with event semantics to explicitly strengthen the correspondence between semantics and temporal events and improve span-level separability. Local Relational Consistency Alignment (LRCA) constructs a textual relation matrix derived from clip-level captions and aligns it with the temporal feature similarity matrix in the model, enhancing temporal consistency while capturing local structural information. MASRA includes two simple supporting modules, semantic-guided enhancement and second-order relational attention, to better utilize the learned semantic context and relational structure. Moreover, we introduce Decoupled Alignment Interaction (DAI) with a context-aware codebook to adaptively absorb query-irrelevant semantics and alleviate the cross-modal gap. The MLLM is only invoked during training and is not used at inference. Extensive experiments show that MASRA outperforms existing methods, and ablation studies validate its effectiveness.

Related papers

• STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition [23.546777614096424]
  Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples.<n>Despite progress with vision-language models, existing approaches still suffer from semantic-temporal misalignment.<n>We propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework consisting of a semantic-alignment component and a temporal-aware component.
  arXiv  Detail & Related papers  (2026-05-13T08:54:38Z)
• Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding [6.673833357112721]
  textbfBridge-STG is an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence.<n>Bridge-STG achieves state-of-the-art performance among MLLM-based methods.
  arXiv  Detail & Related papers  (2026-04-09T09:14:00Z)
• From Consistency to Complementarity: Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning [12.903267405917388]
  We propose MADI, a multi-modal large language model (MLLM) enhanced with fine-grained alignment and disentangled interaction.<n>Experiments on synthetic and real-world benchmarks show that MADI consistently outperforms general-purpose LLMs and time-series-specialized MLLMs.
  arXiv  Detail & Related papers  (2026-01-29T09:13:46Z)
• 1 + 1 > 2: Detector-Empowered Video Large Language Model for Spatio-Temporal Grounding and Reasoning [53.28271278708241]
  We present a Detector-Empowered Video LLM, short for DEViL.<n> DEViL couples a Video LLM with an open-vocabulary detector (OVD)<n>Unlike tokens that merely serve as spatial prompts or segmentor switches, the RST functions as both a control signal and a replacement for the OVD's text embedding.
  arXiv  Detail & Related papers  (2025-12-07T06:11:15Z)
• Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment [15.51412479114864]
  We introduce MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that guides cross modal representation learning.<n>MaPLE formulates the learning process as reinforcement learning with two key components: automatic preference data construction using off-the-shelf MLLM, and a new Relative Preference Alignment (RPA) loss.<n> Experimental results show that our preference-guided alignment achieves substantial gains in fine-grained cross-modal retrieval.
  arXiv  Detail & Related papers  (2025-06-08T02:33:35Z)
• LLaFEA: Frame-Event Complementary Fusion for Fine-Grained Spatiotemporal Understanding in LMMs [55.81291976637705]
  Large models (LMMs) excel in scene understanding but struggle with fine-temporal reasoning due to weak alignment between linguistic and visual representations.<n>Existing methods map textual positions and durations into the visual space from frame-based videos, but suffer from temporal sparsity that limits temporal coordination.<n>We introduce LFEA to leverage event cameras for temporally dense perception and frame-event fusion.
  arXiv  Detail & Related papers  (2025-03-10T05:30:30Z)
• Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series [3.453940014682793]
  We propose Context-Alignment to align time series (TS) data with a linguistic component in language environments familiar to Large Language Models (LLMs)<n>Such context-level alignment comprises structural alignment and logical alignment, which is achieved by a Dual-Scale Context-Alignment GNNs (DSCA-GNNs)<n>Extensive experiments show the effectiveness of DECA and the importance of Context-Alignment across tasks, particularly in few-shot and zero-shot forecasting.
  arXiv  Detail & Related papers  (2025-01-07T12:40:35Z)
• TableTime: Reformulating Time Series Classification as Training-Free Table Understanding with Large Language Models [20.077169400691435]
  Large language models (LLMs) have demonstrated their effectiveness in multivariate time series classification.<n>LLMs directly encode embeddings for time series within the latent space of LLMs from scratch to align with semantic space of LLMs.<n>We propose TableTime, which reformulates MTSC as a table understanding task.
  arXiv  Detail & Related papers  (2024-11-24T07:02:32Z)
• Guiding the PLMs with Semantic Anchors as Intermediate Supervision: Towards Interpretable Semantic Parsing [57.11806632758607]
  We propose to incorporate the current pretrained language models with a hierarchical decoder network. By taking the first-principle structures as the semantic anchors, we propose two novel intermediate supervision tasks. We conduct intensive experiments on several semantic parsing benchmarks and demonstrate that our approach can consistently outperform the baselines.
  arXiv  Detail & Related papers  (2022-10-04T07:27:29Z)
• Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization [87.47977407022492]
  This paper argues that learning by contextually comparing sequence-to-sequence distinctions offers an essential inductive bias in weakly-supervised action localization. Under a differentiable dynamic programming formulation, two complementary contrastive objectives are designed, including Fine-grained Sequence Distance (FSD) contrasting and Longest Common Subsequence (LCS) contrasting. Our method achieves state-of-the-art performance on two popular benchmarks.
  arXiv  Detail & Related papers  (2022-03-31T05:13:50Z)
• Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding [53.377028000325424]
  We propose an Iterative Alignment Network (IA-Net) for temporal sentence grounding task. We pad multi-modal features with learnable parameters to alleviate the nowhere-to-attend problem of non-matched frame-word pairs. We also devise a calibration module following each attention module to refine the alignment knowledge.
  arXiv  Detail & Related papers  (2021-09-14T02:08:23Z)
• Weakly Supervised Temporal Adjacent Network for Language Grounding [96.09453060585497]
  We introduce a novel weakly supervised temporal adjacent network (WSTAN) for temporal language grounding. WSTAN learns cross-modal semantic alignment by exploiting temporal adjacent network in a multiple instance learning (MIL) paradigm. An additional self-discriminating loss is devised on both the MIL branch and the complementary branch, aiming to enhance semantic discrimination by self-supervising.
  arXiv  Detail & Related papers  (2021-06-30T15:42:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.