OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Models

• URL: http://arxiv.org/abs/2509.17627v1
• Date: Mon, 22 Sep 2025 11:35:55 GMT
• Title: OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Models
• Abstract summary: We focus on the task of Mask-free Video Insertion and aim to resolve three key challenges: data scarcity, subject-scene equilibrium, and insertion harmonization.<n>To address the data scarcity, we propose a new data pipeline InsertPipe, constructing diverse cross-pair data automatically.<n>We develop OmniInsert, a novel unified framework for mask-free video insertion from both single and multiple subject references.
• Score: 27.792199133175718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in video insertion based on diffusion models are impressive. However, existing methods rely on complex control signals but struggle with subject consistency, limiting their practical applicability. In this paper, we focus on the task of Mask-free Video Insertion and aim to resolve three key challenges: data scarcity, subject-scene equilibrium, and insertion harmonization. To address the data scarcity, we propose a new data pipeline InsertPipe, constructing diverse cross-pair data automatically. Building upon our data pipeline, we develop OmniInsert, a novel unified framework for mask-free video insertion from both single and multiple subject references. Specifically, to maintain subject-scene equilibrium, we introduce a simple yet effective Condition-Specific Feature Injection mechanism to distinctly inject multi-source conditions and propose a novel Progressive Training strategy that enables the model to balance feature injection from subjects and source video. Meanwhile, we design the Subject-Focused Loss to improve the detailed appearance of the subjects. To further enhance insertion harmonization, we propose an Insertive Preference Optimization methodology to optimize the model by simulating human preferences, and incorporate a Context-Aware Rephraser module during reference to seamlessly integrate the subject into the original scenes. To address the lack of a benchmark for the field, we introduce InsertBench, a comprehensive benchmark comprising diverse scenes with meticulously selected subjects. Evaluation on InsertBench indicates OmniInsert outperforms state-of-the-art closed-source commercial solutions. The code will be released.

Related papers

• Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework [52.477407591581176]
  Mask-free video object insertion has emerged as a challenging task, requiring harmonious integration of reference objects into source videos.<n>We propose textittextbfSmart-Insertion-V, an end-to-end textbfDual-Stream framework that concurrently conducts video insertion and image style transfer.
  arXiv  Detail & Related papers  (2026-05-22T17:54:54Z)
• AlcheMinT: Fine-grained Temporal Control for Multi-Reference Consistent Video Generation [58.844504598618094]
  We propose AlcheMinT, a unified framework that introduces explicit timestamps conditioning for subject-driven video generation.<n>Our approach introduces a novel positional encoding mechanism that unlocks the encoding of temporal intervals, associated in our case with subject identities.<n>We incorporate subject-descriptive text tokens to strengthen binding between visual identity and video captions, mitigating ambiguity during generation.
  arXiv  Detail & Related papers  (2025-12-11T18:59:34Z)
• Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence [81.82643953694485]
  We present FRESCO, which integrates intra-frame correspondence with inter-frame correspondence to formulate a more robust spatial-temporal constraint.<n>Our method goes beyond attention guidance to explicitly optimize features, achieving high spatial-temporal consistency with the input video.<n>We verify FRESCO adaptations on two zero-shot tasks of video-to-video translation and text-guided video editing.
  arXiv  Detail & Related papers  (2025-12-03T15:51:11Z)
• CloseUpShot: Close-up Novel View Synthesis from Sparse-views via Point-conditioned Diffusion Model [50.93869080795228]
  Reconstructing 3D scenes and synthesizing novel views from sparse input views is a highly challenging task.<n>Recent advances in video diffusion models have demonstrated strong temporal reasoning capabilities.<n>We present a diffusion-based framework, called CloseUpShot, for close-up novel view synthesis from sparse inputs via point-conditioned video diffusion.
  arXiv  Detail & Related papers  (2025-11-17T08:20:06Z)
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces [10.85468238780625]
  We propose a novel framework for building multimodal diffusion models on arbitrary state spaces.<n>By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously.
  arXiv  Detail & Related papers  (2025-06-09T16:20:20Z)
• MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement [47.064467920954776]
  We introduce MAGREF, a unified and effective framework for any-reference video generation.<n>Our approach incorporates masked guidance and a subject disentanglement mechanism.<n>Experiments on a comprehensive benchmark demonstrate that MAGREF consistently outperforms existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-05-29T17:58:15Z)
• Appearance-Based Refinement for Object-Centric Motion Segmentation [85.2426540999329]
  We introduce an appearance-based refinement method that leverages temporal consistency in video streams to correct inaccurate flow-based proposals. Our approach involves a sequence-level selection mechanism that identifies accurate flow-predicted masks as exemplars. Its performance is evaluated on multiple video segmentation benchmarks, including DAVIS, YouTube, SegTrackv2, and FBMS-59.
  arXiv  Detail & Related papers  (2023-12-18T18:59:51Z)
• MagDiff: Multi-Alignment Diffusion for High-Fidelity Video Generation and Editing [90.06041718086317]
  We propose a unified Multi-alignment Diffusion, dubbed as MagDiff, for both tasks of high-fidelity video generation and editing. The proposed MagDiff introduces three types of alignments, including subject-driven alignment, adaptive prompts alignment, and high-fidelity alignment.
  arXiv  Detail & Related papers  (2023-11-29T03:36:07Z)
• Conditional Modeling Based Automatic Video Summarization [70.96973928590958]
  The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story. Video summarization methods rely on visual factors, such as visual consecutiveness and diversity, which may not be sufficient to fully understand the content of the video. A new approach to video summarization is proposed based on insights gained from how humans create ground truth video summaries.
  arXiv  Detail & Related papers  (2023-11-20T20:24:45Z)
• CCVS: Context-aware Controllable Video Synthesis [95.22008742695772]
  presentation introduces a self-supervised learning approach to the synthesis of new video clips from old ones. It conditions the synthesis process on contextual information for temporal continuity and ancillary information for fine control.
  arXiv  Detail & Related papers  (2021-07-16T17:57:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.