Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework

• URL: http://arxiv.org/abs/2605.23891v1
• Date: Fri, 22 May 2026 17:54:54 GMT
• Title: Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework
• Abstract summary: Mask-free video object insertion has emerged as a challenging task, requiring harmonious integration of reference objects into source videos.<n>We propose textittextbfSmart-Insertion-V, an end-to-end textbfDual-Stream framework that concurrently conducts video insertion and image style transfer.
• Score: 52.477407591581176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mask-free video object insertion has emerged as a challenging task, requiring harmonious integration of reference objects into source videos. However, existing methods struggle when references exhibit severe stylistic domain gaps with the source scene. To overcome this, we propose \textit{\textbf{Smart-Insertion-V}}, an end-to-end \textbf{Dual-Stream} framework that concurrently conducts video insertion and image style transfer. Within this framework, the image stream synchronously guides the video generation process, while a \textbf{Closed-loop Feedback} mechanism is further incorporated to ensure robust insertion. Inevitably, integrating these diverse conditioning signals results in feature entanglement and style leakage. To tackle this issue, we design \textbf{Dual-World-View RoPE} to distinguish different signals via spatial-temporal offsets without incurring heavy training overhead. Furthermore, to facilitate spatial grounding and stylistic adaptation, we introduce a \textbf{Decoupled Guidance Module} that leverages a Vision-Language Model for semantic reasoning while preserving original temporal guidance with native text encoder. To bridge data gap for harmonious reference insertion task, we propose a data curation pipeline and will release an \textbf{open-source dataset}. Experiments demonstrate that our method can insert objects into plausible positions while achieving the most harmonious results.

Related papers

• Imagine Before You Draw: Visual Prompt Engineering for Image Generation [67.81347924426714]
  We propose Visual Prompt Engineering (VPE), which can be seamlessly integrated into internal frameworks.<n>We validate VPE across class-conditional generation, text-to-image generation, and image editing.<n>Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration, achieve substantially better editing preservation.
  arXiv  Detail & Related papers  (2026-06-03T05:01:36Z)
• Controllable Video Object Insertion via Multiview Priors [30.363004603304418]
  Video object insertion is a critical task for dynamically inserting new objects into existing environments.<n>We propose a novel solution for Video Object Insertion, which integrates multi-view object priors.<n>We show that our solution significantly improves the quality of video object insertion, providing stable and realistic integration.
  arXiv  Detail & Related papers  (2026-04-16T02:39:15Z)
• Gloria: Consistent Character Video Generation via Content Anchors [63.7455328319917]
  We propose representing character visual attributes through a compact set of anchor frames.<n>This design provides stable references for consistency, while reference-based video generation faces challenges of copy-pasting and multi-reference conflicts.<n> Experiments show our method generates high-quality character videos exceeding 10 minutes, and achieves expressive identity and appearance consistency across views, surpassing existing methods.
  arXiv  Detail & Related papers  (2026-03-31T16:06:16Z)
• ContextAnyone: Context-Aware Diffusion for Character-Consistent Text-to-Video Generation [36.29956463871403]
  Text-to-video (T2V) generation has advanced rapidly, yet maintaining consistent character identities across scenes remains a major challenge.<n>We propose textbfContextAnyone, a context-aware diffusion framework that achieves character-consistent video generation from text and a single reference image.<n>Our method jointly reconstructs the reference image and generates new video frames, enabling the model to fully perceive and utilize reference information.
  arXiv  Detail & Related papers  (2025-12-08T09:12:18Z)
• CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion [62.04833878126661]
  We tackle the dual challenges of video understanding and controllable video generation within a unified diffusion framework.<n>We propose CtrlVDiff, a unified diffusion model trained with a Hybrid Modality Control Strategy (HMCS) that routes and fuses features from depth, normals, segmentation, edges, and graphics-based intrinsics (albedo, roughness, metallic)<n>Across understanding and generation benchmarks, CtrlVDiff delivers superior controllability and fidelity, enabling layer-wise edits (relighting, material adjustment, object insertion) and surpassing state-of-the-art baselines while remaining robust when some modalities are unavailable.
  arXiv  Detail & Related papers  (2025-11-26T07:27:11Z)
• OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Models [27.792199133175718]
  We focus on the task of Mask-free Video Insertion and aim to resolve three key challenges: data scarcity, subject-scene equilibrium, and insertion harmonization.<n>To address the data scarcity, we propose a new data pipeline InsertPipe, constructing diverse cross-pair data automatically.<n>We develop OmniInsert, a novel unified framework for mask-free video insertion from both single and multiple subject references.
  arXiv  Detail & Related papers  (2025-09-22T11:35:55Z)
• SAGA: Learning Signal-Aligned Distributions for Improved Text-to-Image Generation [9.212970624261272]
  State-of-the-art text-to-image models produce visually impressive results but often struggle with precise alignment to text prompts.<n>We propose a novel approach that learns a high-success-rate distribution conditioned on a target prompt.<n>Our method explicitly models the signal component during the denoising process, offering fine-grained control that mitigates over-optimization.
  arXiv  Detail & Related papers  (2025-08-19T14:31:15Z)
• MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement [47.064467920954776]
  We introduce MAGREF, a unified and effective framework for any-reference video generation.<n>Our approach incorporates masked guidance and a subject disentanglement mechanism.<n>Experiments on a comprehensive benchmark demonstrate that MAGREF consistently outperforms existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-05-29T17:58:15Z)
• Insert Anything: Image Insertion via In-Context Editing in DiT [19.733787045511775]
  We present a unified framework for reference-based image insertion that seamlessly integrates objects from reference images into target scenes under flexible, user-specified control guidance.<n>Our approach is trained once on our new AnyInsertion dataset--comprising 120K prompt-image pairs covering diverse tasks such as person, object, and garment insertion--and effortlessly generalizes to a wide range of insertion scenarios.
  arXiv  Detail & Related papers  (2025-04-21T10:19:12Z)
• Semantic Lens: Instance-Centric Semantic Alignment for Video Super-Resolution [36.48329560039897]
  inter-frame alignment is a critical clue of video super-resolution (VSR) We introduce a novel paradigm for VSR named Semantic Lens. Video is modeled as instances, events, and scenes via a Semantic Extractor.
  arXiv  Detail & Related papers  (2023-12-13T01:16:50Z)
• UpFusion: Novel View Diffusion from Unposed Sparse View Observations [66.36092764694502]
  UpFusion can perform novel view synthesis and infer 3D representations for an object given a sparse set of reference images. We show that this mechanism allows generating high-fidelity novel views while improving the synthesis quality given additional (unposed) images.
  arXiv  Detail & Related papers  (2023-12-11T18:59:55Z)
• LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [121.45667242282721]
  We propose a coarse-to-fine paradigm to achieve layout planning and image generation. Our proposed method outperforms the state-of-the-art models in terms of photorealistic layout and image generation.
  arXiv  Detail & Related papers  (2023-08-09T17:45:04Z)
• Redundancy-aware Transformer for Video Question Answering [71.98116071679065]
  We propose a novel transformer-based architecture, that aims to model VideoQA in a redundancy-aware manner. To address the neighboring-frame redundancy, we introduce a video encoder structure that emphasizes the object-level change in neighboring frames. As for the cross-modal redundancy, we equip our fusion module with a novel adaptive sampling, which explicitly differentiates the vision-language interactions.
  arXiv  Detail & Related papers  (2023-08-07T03:16:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.