ROSE: Remove Objects with Side Effects in Videos

• URL: http://arxiv.org/abs/2508.18633v1
• Date: Tue, 26 Aug 2025 03:18:31 GMT
• Title: ROSE: Remove Objects with Side Effects in Videos
• Authors: Chenxuan Miao, Yutong Feng, Jianshu Zeng, Zixiang Gao, Hantang Liu, Yunfeng Yan, Donglian Qi, Xi Chen, Bin Wang, Hengshuang Zhao,
• Abstract summary: ROSE is a framework that studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror.<n>To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing.<n>Additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos.
• Score: 47.196851721000655
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video object removal has achieved advanced performance due to the recent success of video generative models. However, when addressing the side effects of objects, e.g., their shadows and reflections, existing works struggle to eliminate these effects for the scarcity of paired video data as supervision. This paper presents ROSE, termed Remove Objects with Side Effects, a framework that systematically studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror. Given the challenges of curating paired videos exhibiting the aforementioned effects, we leverage a 3D rendering engine for synthetic data generation. We carefully construct a fully-automatic pipeline for data preparation, which simulates a large-scale paired dataset with diverse scenes, objects, shooting angles, and camera trajectories. ROSE is implemented as an video inpainting model built on diffusion transformer. To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing. Moreover, additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos. To fully investigate the model performance on various side effect removal, we presents a new benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five special side effects for comprehensive evaluation. Experimental results demonstrate that ROSE achieves superior performance compared to existing video object erasing models and generalizes well to real-world video scenarios. The project page is https://rose2025-inpaint.github.io/.

Related papers

• Tuning-free Visual Effect Transfer across Videos [91.93897438317397]
  RefVFX is a framework that transfers complex temporal effects from a reference video onto a target video or image in a feed-forward manner.<n>We introduce a large-scale dataset of triplets, where each triplet consists of a reference effect video, an input image or video, and a corresponding output video.<n>We show that RefVFX produces visually consistent and temporally coherent edits, generalizes across unseen effect categories, and outperforms prompt-only baselines in both quantitative metrics and human preference.
  arXiv  Detail & Related papers  (2026-01-12T18:59:32Z)
• Over++: Generative Video Compositing for Layer Interaction Effects [9.071187452755206]
  We present Over++, a video effect generation framework that makes no assumptions about camera pose, scene stationarity, or depth supervision.<n>Despite training on limited data, Over++ produces diverse and realistic environmental effects and outperforms existing baselines in effect generation and scene preservation.
  arXiv  Detail & Related papers  (2025-12-22T18:39:58Z)
• ObjectMover: Generative Object Movement with Video Prior [69.75281888309017]
  We present ObjectMover, a generative model that can perform object movement in challenging scenes.<n>We show that with this approach, our model is able to adjust to complex real-world scenarios.<n>We propose a multi-task learning strategy that enables training on real-world video data to improve the model generalization.
  arXiv  Detail & Related papers  (2025-03-11T04:42:59Z)
• OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Data [21.469971783624402]
  In this paper, we propose Video4Removal, a large-scale dataset comprising over 100,000 high-quality samples with realistic object shadows and reflections.<n>By constructing object-background pairs from video frames with off-the-shelf vision models, the labor costs of data acquisition can be significantly reduced.<n>To avoid generating shape-like artifacts and unintended content, we propose Object-Background Guidance.<n>We present OmniEraser, a novel method that seamlessly removes objects and their visual effects using only object masks as input.
  arXiv  Detail & Related papers  (2025-01-13T15:12:40Z)
• WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction [82.79642869586587]
  WALDO is a novel approach to the prediction of future video frames from past ones. Individual images are decomposed into multiple layers combining object masks and a small set of control points. The layer structure is shared across all frames in each video to build dense inter-frame connections.
  arXiv  Detail & Related papers  (2022-11-25T18:59:46Z)
• Occlusion-Aware Video Object Inpainting [72.38919601150175]
  This paper presents occlusion-aware video object inpainting, which recovers both the complete shape and appearance for occluded objects in videos. Our technical contribution VOIN jointly performs video object shape completion and occluded texture generation. For more realistic results, VOIN is optimized using both T-PatchGAN and a newoc-temporal YouTube attention-based multi-class discriminator.
  arXiv  Detail & Related papers  (2021-08-15T15:46:57Z)
• Omnimatte: Associating Objects and Their Effects in Video [100.66205249649131]
  Scene effects related to objects in video are typically overlooked by computer vision. In this work, we take a step towards solving this novel problem of automatically associating objects with their effects in video. Our model is trained only on the input video in a self-supervised manner, without any manual labels, and is generic---it produces omnimattes automatically for arbitrary objects and a variety of effects.
  arXiv  Detail & Related papers  (2021-05-14T17:57:08Z)
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces [77.07767833443256]
  We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. In contrast to state-of-the-art methods in object-centric generative modeling, RELATE also extends naturally to dynamic scenes and generates videos of high visual fidelity.
  arXiv  Detail & Related papers  (2020-07-02T17:27:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.