SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models

• URL: http://arxiv.org/abs/2605.07800v1
• Date: Fri, 08 May 2026 14:36:32 GMT
• Title: SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models
• Abstract summary: Recent diffusion models (VDMs) synthesize visually convincing clips, yet still drop entities, mis-bind attributes, weaken the interactions specified in the prompt-alignment rubric.<n>We present SARA, Adaptive Representationally Alignment, which keeps token-relation on a frozen VFM target and adds a text-conditioned saliency that decides which token pairs carry supervision.
• Score: 24.580427340782524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent video diffusion models (VDMs) synthesize visually convincing clips, yet still drop entities, mis-bind attributes, and weaken the interactions specified in the prompt. Representation-alignment objectives such as VideoREPA and MoAlign improve fine-grained text following by distilling spatio-temporal token relations from a frozen visual foundation model, but their pairwise supervision budget is allocated by visual or motion cues rather than by how relevant each pair is to the prompt. We present SARA, Semantically Adaptive Relational Alignment, which keeps token-relation distillation (TRD) on a frozen VFM target and adds a text-conditioned saliency that decides which token pairs carry supervision. A lightweight Stage 1 aligner is trained with per-entity SAM 3.1 mask supervision and an InfoNCE regulariser, and its continuous saliency is fused into TRD through a pair-routing operator that assigns each token pair a weight whenever either of its two endpoints is salient, thereby routing supervision toward subject-subject and subject-background pairs and away from background-background ones. In the Wan2.2 continual-training setting, SARA improves both text alignment and motion quality over SFT, VideoREPA, and MoAlign on a 13-dimension VLM rubric, on the public VBench benchmarks, and in a blind user study.

Related papers

• DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding [11.584150931573738]
  Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries.<n>Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations.<n>We propose DSE-VTG, a framework that addresses both frame-wise visual encoding and fixed query embeddings.
  arXiv  Detail & Related papers  (2026-09-08T15:03:41Z)
• Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos [70.78278988781882]
  We present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs.<n>Our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating.
  arXiv  Detail & Related papers  (2026-05-25T18:02:20Z)
• Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing [55.211537893248675]
  We propose RVEDiT, an implicit Reasoning Video Editing DiT framework built around two complementary components.<n> RVEDiT consistently outperforms state-of-the-art baselines, with particularly strong gains on localized and compositional edits.
  arXiv  Detail & Related papers  (2026-05-23T17:22:14Z)
• Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning [33.42009265471664]
  Weakly-supervised video scene graph generation (WS-VSGG) aims to parse video content into structured relational triplets without bounding box annotations and with only sparse temporal labeling.<n>We introduce a learnable pair affinity that estimates the likelihood of interaction between subject-object pairs.<n>Our approach consistently yields substantial improvements across different baselines and backbones, achieving state-of-the-art WS-VSGG performance.
  arXiv  Detail & Related papers  (2026-03-23T04:28:51Z)
• SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing [76.349958946335]
  We present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling.<n>First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames.<n>Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment [32.178798481694834]
  We introduce MoBind, a hierarchical contrastive learning framework designed to address three challenges.<n>To isolate motion-relevant cues, MoBind aligns IMU signals with skeletal motion sequences rather than raw pixels.<n>To capture detailed temporal correspondence, MoBind employs a hierarchical contrastive strategy that first aligns token-level temporal segments, then fuses local (body-part) alignment with global (body-wide) motion aggregation.
  arXiv  Detail & Related papers  (2026-02-22T01:54:29Z)
• Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models [41.59364061354628]
  Image-to-Video (I2V) generation aims to synthesize a video from a reference image and a text prompt.<n>Existing I2V models prioritize visual consistency.<n>How to effectively couple this dual guidance to ensure strong adherence to the text prompt remains underexplored.
  arXiv  Detail & Related papers  (2026-01-12T07:48:26Z)
• Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing [26.317163478761916]
  Weakly-supervised audio-visual video parsing seeks to detect audible, visible, and audio-visual events without temporal annotations.<n>We propose an exponential moving average (EMA)-guided pseudo supervision framework that generates reliable segment-level masks.<n>We also propose a class-aware cross-modal agreement (CMA) loss that aligns audio and visual embeddings at reliable segment-class pairs.
  arXiv  Detail & Related papers  (2025-09-17T15:38:05Z)
• Video-LLMs with Temporal Visual Screening [59.18455762289321]
  Temporal Visual Screening (TVS) is a new task that universally pre-processes video question answering and instruction tuning data.<n>TVS is formulated as a modular front-end adapter task that can be seamlessly integrated into both Video Instruction Tuning (training) and Video Question Answering (inference) pipelines.<n> Experiments demonstrate that incorporating TVS yields relative gains of 7.33% (training) and 34.6% (inference)
  arXiv  Detail & Related papers  (2025-08-27T14:33:32Z)
• Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models [31.138079872368532]
  Fine-tuning Video Diffusion Models (VDMs) at the user level to generate videos that reflect specific attributes of training data presents notable challenges.<n>Recent work such as Representation Alignment (REPA) has shown promise in improving the convergence and quality of DiT-based image diffusion models.<n>We introduce Cross-frame Representation Alignment (CREPA), a novel regularization technique that aligns hidden states of a frame with external features from neighboring frames.
  arXiv  Detail & Related papers  (2025-06-10T20:34:47Z)
• Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment [71.16699226211504]
  We propose to learn fine-grained action features that are invariant to the viewpoints by aligning egocentric and exocentric videos in time. To this end, we propose AE2, a self-supervised embedding approach with two key designs. For evaluation, we establish a benchmark for fine-grained video understanding in the ego-exo context.
  arXiv  Detail & Related papers  (2023-06-08T19:54:08Z)
• Transform-Equivariant Consistency Learning for Temporal Sentence Grounding [66.10949751429781]
  We introduce a novel Equivariant Consistency Regulation Learning framework to learn more discriminative representations for each video. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted. In particular, we devise a self-supervised consistency loss module to enhance the completeness and smoothness of the augmented video.
  arXiv  Detail & Related papers  (2023-05-06T19:29:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.