SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing

• URL: http://arxiv.org/abs/2603.19228v1
• Date: Thu, 19 Mar 2026 17:59:51 GMT
• Title: SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing
• Abstract summary: We present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling.<n>First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames.<n>Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks.
• Score: 76.349958946335
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and datasets will be released.

Related papers

• EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editing [67.99022753510475]
  EditStream is a unified framework for interactive video generation and editing.<n>It unifies multiple video creation and manipulation tasks within a single DiT-based model.<n>It supports Text-to-Video, Image-to-Video, Video-to-Video, Editing propagation, Reference-guided Video Editing, and Camera Pose Change.
  arXiv  Detail & Related papers  (2026-08-16T07:27:18Z)
• UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation [23.675212404889805]
  World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving.<n>Existing cascaded or dual-DiT designs separate video imagination from action prediction.<n>We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Transformer.
  arXiv  Detail & Related papers  (2026-07-06T14:18:18Z)
• Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing [55.211537893248675]
  We propose RVEDiT, an implicit Reasoning Video Editing DiT framework built around two complementary components.<n> RVEDiT consistently outperforms state-of-the-art baselines, with particularly strong gains on localized and compositional edits.
  arXiv  Detail & Related papers  (2026-05-23T17:22:14Z)
• NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing [26.74471251505078]
  NOVA: Sparse Control & Dense Synthesis is a new framework for unpaired video editing.<n>Our experiments demonstrate that NOVA outperforms existing approaches in edit fidelity, motion preservation, and temporal coherence.
  arXiv  Detail & Related papers  (2026-03-03T09:41:06Z)
• PropFly: Learning to Propagate via On-the-Fly Supervision from Pre-trained Video Diffusion Models [35.59605874012795]
  PropFly is a training pipeline for propagation-based video editing.<n>PropFly relies on pre-trained video diffusion models (VDMs) instead of requiring off-the-shelf or precomputed paired video editing datasets.<n>Our pipeline enables an adapter attached to the pre-trained VDM to learn to propagate edits via Guidance-Modulated Flow Matching (GMFM) loss.
  arXiv  Detail & Related papers  (2026-02-24T06:11:08Z)
• CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion [62.04833878126661]
  We tackle the dual challenges of video understanding and controllable video generation within a unified diffusion framework.<n>We propose CtrlVDiff, a unified diffusion model trained with a Hybrid Modality Control Strategy (HMCS) that routes and fuses features from depth, normals, segmentation, edges, and graphics-based intrinsics (albedo, roughness, metallic)<n>Across understanding and generation benchmarks, CtrlVDiff delivers superior controllability and fidelity, enabling layer-wise edits (relighting, material adjustment, object insertion) and surpassing state-of-the-art baselines while remaining robust when some modalities are unavailable.
  arXiv  Detail & Related papers  (2025-11-26T07:27:11Z)
• Show Me: Unifying Instructional Image and Video Generation with Diffusion Models [16.324312147741495]
  We propose a unified framework that enables image manipulation and video prediction.<n>We introduce structure and motion consistency rewards to improve structural fidelity and temporal coherence.<n> Experiments on diverse benchmarks demonstrate that our method outperforms expert models in both instructional image and video generation.
  arXiv  Detail & Related papers  (2025-11-21T23:24:28Z)
• VALA: Learning Latent Anchors for Training-Free and Temporally Consistent [29.516179213427694]
  We propose VALA, a variational alignment module that adaptively selects key frames and compresses their latent features into semantic anchors for consistent video editing.<n>Our method can be fully integrated into training-free text-to-image based video editing models.
  arXiv  Detail & Related papers  (2025-10-27T03:44:11Z)
• Low-Cost Test-Time Adaptation for Robust Video Editing [4.707015344498921]
  Video editing is a critical component of content creation that transforms raw footage into coherent works aligned with specific visual and narrative objectives.<n>Existing approaches face two major challenges: temporal inconsistencies due to failure in capturing complex motion patterns, and overfitting to simple prompts arising from limitations in UNet backbone architectures.<n>We present Vid-TTA, a lightweight test-time adaptation framework that personalizes optimization for each test video during inference through self-supervised auxiliary tasks.
  arXiv  Detail & Related papers  (2025-07-29T14:31:17Z)
• Motion-Aware Concept Alignment for Consistent Video Editing [57.08108545219043]
  We introduce MoCA-Video (Motion-Aware Concept Alignment in Video), a training-free framework bridging the gap between image-domain semantic mixing and video.<n>Given a generated video and a user-provided reference image, MoCA-Video injects the semantic features of the reference image into a specific object within the video.<n>We evaluate MoCA's performance using the standard SSIM, image-level LPIPS, temporal LPIPS, and introduce a novel metric CASS (Conceptual Alignment Shift Score) to evaluate the consistency and effectiveness of the visual shifts between the source prompt and the modified video frames
  arXiv  Detail & Related papers  (2025-06-01T13:28:04Z)
• COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing [57.76170824395532]
  Video editing is an emerging task, in which most current methods adopt the pre-trained text-to-image (T2I) diffusion model to edit the source video.<n>We propose COrrespondence-guided Video Editing (COVE) to achieve high-quality and consistent video editing.<n>COVE can be seamlessly integrated into the pre-trained T2I diffusion model without the need for extra training or optimization.
  arXiv  Detail & Related papers  (2024-06-13T06:27:13Z)
• Transform-Equivariant Consistency Learning for Temporal Sentence Grounding [66.10949751429781]
  We introduce a novel Equivariant Consistency Regulation Learning framework to learn more discriminative representations for each video. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted. In particular, we devise a self-supervised consistency loss module to enhance the completeness and smoothness of the augmented video.
  arXiv  Detail & Related papers  (2023-05-06T19:29:28Z)
• Understanding Road Layout from Videos as a Whole [82.30800791500869]
  We formulate it as a top-view road attributes prediction problem and our goal is to predict these attributes for each frame both accurately and consistently. We exploit the following three novel aspects: leveraging camera motions in videos, including context cuesand incorporating long-term video information.
  arXiv  Detail & Related papers  (2020-07-02T00:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.