BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

• URL: http://arxiv.org/abs/2608.20107v1
• Date: Thu, 20 Aug 2026 14:37:40 GMT
• Title: BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal
• Abstract summary: We introduce BeyondMasks, a paired benchmark for causally consistent video object removal.<n>The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions.<n>We propose CORE, a structured vision language model based evaluation protocol.
• Score: 27.338971504674983
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on masked region fidelity, treating removal as local inpainting. In real scenes, object removal is a causal intervention: eliminating an object also requires removing its induced physical effects, such as shadows, reflections, illumination changes, translucency, and dynamic traces. Existing benchmarks lack aligned clean references or remain limited to simplified synthetic settings, preventing systematic evaluation of causal consistency. We introduce BeyondMasks, a paired benchmark for causally consistent video object removal, consisting of temporally aligned synthetic and real world video pairs with clean background references. The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions, and supports both mask based and instruction driven editing. We further propose CORE, a structured vision language model based evaluation protocol that jointly measures object disappearance and after effect consistency, aligning more closely with human judgments than existing metrics. Benchmarking state of the art methods reveals systematic failures in removing secondary physical effects despite high masked region fidelity, exposing a gap between visual plausibility and causal correctness. BeyondMasks reframes video object removal as causal scene consistency rather than local reconstruction and provides a unified framework for its evaluation.

Related papers

• MemoBench: Benchmarking World Modeling in Dynamically Changing Environments [72.23517478870605]
  We introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments.<n>We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment.
  arXiv  Detail & Related papers  (2026-06-25T20:37:39Z)
• Physics-Aware Video Instance Removal Benchmark [23.60215680694382]
  Video Instance Removal (VIR) requires removing target objects while maintaining background integrity and physical consistency.<n>Current benchmarks primarily assess visual plausibility, often overlooking the physical causalities, such as lingering shadows.<n>We introduce the Physics-Aware Video Instance Removal benchmark, featuring 95 high-quality videos annotated with instance-accurate masks and removal prompts.
  arXiv  Detail & Related papers  (2026-04-07T14:02:53Z)
• VOID: Video Object and Interaction Deletion [50.155301620235285]
  Existing video object removal methods excel at inpainting content "behind" the object and correcting appearance-level artifacts such as shadows and reflections.<n>We present VOID, a video object removal framework designed to perform physically-plausible inpainting in complex scenarios.
  arXiv  Detail & Related papers  (2026-04-02T17:36:53Z)
• From Understanding to Erasing: Towards Complete and Stable Video Object Removal [8.035439287403983]
  Video object removal aims to eliminate target objects from videos while plausibly completing missing regions and preservingtemporal consistency.<n>It remains challenging to remove object-induced side effects without compromising overall coherence.<n>This paper introduces understanding into erasing from two complementary perspectives.
  arXiv  Detail & Related papers  (2026-04-02T06:50:39Z)
• EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing [50.43992550991499]
  Video object removal aims to eliminate dynamic target objects and their visual effects, such as deformation, shadows, and reflections, while restoring seamless backgrounds.<n>Recent diffusion-based video inpainting and object removal methods can remove the objects but often struggle to erase these effects and to synthesize coherent backgrounds.<n>We introduce VOR (Video Object Removal), a large-scale dataset that provides diverse paired videos.<n>We propose EffectErase, an effect-aware video object removal method that treats video object insertion as the inverse auxiliary task within a reciprocal learning scheme.
  arXiv  Detail & Related papers  (2026-03-19T17:59:22Z)
• GeoRemover: Removing Objects and Their Causal Visual Artifacts [38.57204197752552]
  Object removal should eliminate both the target object and its causal visual artifacts, such as shadows and reflections.<n>We propose a geometry-aware two-stage framework that decouples object removal into (1) geometry removal and (2) appearance rendering.<n>Our method achieves state-of-the-art performance in removing both objects and their associated artifacts on two popular benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T02:04:19Z)
• ROSE: Remove Objects with Side Effects in Videos [47.196851721000655]
  ROSE is a framework that studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror.<n>To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing.<n>Additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos.
  arXiv  Detail & Related papers  (2025-08-26T03:18:31Z)
• Discovering Objects that Can Move [55.743225595012966]
  We study the problem of object discovery -- separating objects from the background without manual labels. Existing approaches utilize appearance cues, such as color, texture, and location, to group pixels into object-like regions. We choose to focus on dynamic objects -- entities that can move independently in the world.
  arXiv  Detail & Related papers  (2022-03-18T21:13:56Z)
• Weakly-Supervised Video Object Grounding via Causal Intervention [82.68192973503119]
  We target at the task of weakly-supervised video object grounding (WSVOG), where only video-sentence annotations are available during model learning. It aims to localize objects described in the sentence to visual regions in the video, which is a fundamental capability needed in pattern analysis and machine learning.
  arXiv  Detail & Related papers  (2021-12-01T13:13:03Z)
• Occlusion-Aware Video Object Inpainting [72.38919601150175]
  This paper presents occlusion-aware video object inpainting, which recovers both the complete shape and appearance for occluded objects in videos. Our technical contribution VOIN jointly performs video object shape completion and occluded texture generation. For more realistic results, VOIN is optimized using both T-PatchGAN and a newoc-temporal YouTube attention-based multi-class discriminator.
  arXiv  Detail & Related papers  (2021-08-15T15:46:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.