SceneForge: Structured World Supervision from 3D Interventions

• URL: http://arxiv.org/abs/2605.14399v1
• Date: Thu, 14 May 2026 05:38:00 GMT
• Title: SceneForge: Structured World Supervision from 3D Interventions
• Abstract summary: multimodal learning tasks require supervision that remains consistent across edits, viewpoints, and scene-level interventions.<n>We present SceneForge, an intervention-driven framework that generates structured supervision from editable 3D world states.
• Score: 5.973748478214713
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many multimodal learning tasks require supervision that remains consistent across edits, viewpoints, and scene-level interventions. However, such supervision is difficult to obtain from observation-level datasets, which do not expose the underlying scene state or how changes propagate through it. We present SceneForge, an intervention-driven framework that generates structured supervision from editable 3D world states. SceneForge represents each scene as a persistent world with semantic, geometric, and physical dependencies. By applying explicit interventions (e.g., object removal or camera variation) and propagating their effects through scene dependencies, SceneForge renders supervision that remains consistent with object structure and scene-level effects. This produces aligned outputs including counterfactual observations, multi-view observations, and effect-aware signals such as shadows and reflections, all derived from a shared world state rather than post hoc image-space processing. We instantiate SceneForge using Infinigen and Blender to construct a licensing-clean indoor supervision resource with a large number of counterfactual pairs and aligned annotations from over 2K scenes, covering both diverse single-view and registered multi-view settings. Under matched training budgets, incorporating SceneForge supervision improves both object removal and scene removal performance across multiple benchmarks in both quantitative and qualitative evaluation. These results indicate that modeling supervision as structured state transitions in editable worlds provides a practical and scalable foundation for intervention-consistent multimodal learning.

Related papers

• ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures [18.468596508491967]
  ReScene is a framework that threads multi-view geometry throughout the pipeline as a unifying prior.<n>Our method consists of two main components: HierView prioritizes reconstruction views based on semantic consistency and 3D coverage.<n>ReScene sets a new state of the art across geometry, rendering, and perceptual quality on a set of ScanNet scenes.
  arXiv  Detail & Related papers  (2026-06-26T13:08:40Z)
• SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestration [32.39337008619354]
  We propose a multi-agent orchestration framework that decomposes single-image 3D scene generation into three structured stages.<n>We introduce a geometry-aware layout predictor supervised by sparse geometric priors derived from point maps.<n>Our method consistently outperforms prior approaches in geometric accuracy, spatial consistency, and perceptual realism.
  arXiv  Detail & Related papers  (2026-06-07T01:38:39Z)
• Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion [69.64308172688368]
  Tri-Prompting is a unified framework that integrates scene composition, multi-view subject consistency, and motion control.<n>Results demonstrate that Tri-Prompting significantly outperforms specialized baselines such as Phantom and DaS in multi-view subject identity, 3D consistency, and motion accuracy.
  arXiv  Detail & Related papers  (2026-03-16T17:59:05Z)
• Hierarchical Image-Guided 3D Point Cloud Segmentation in Industrial Scenes via Multi-View Bayesian Fusion [4.679314646805623]
  3D segmentation is critical for understanding complex scenes with dense layouts and multi-scale objects.<n>Existing 3D point-based methods require costly annotations, while image-guided methods often suffer from semantic inconsistencies across views.<n>We propose a hierarchical image-guided 3D segmentation framework that progressively refines segmentation from instance-level to part-level.
  arXiv  Detail & Related papers  (2025-12-07T15:15:52Z)
• Object-level Scene Deocclusion [92.39886029550286]
  We present a new self-supervised PArallel visible-to-COmplete diffusion framework, named PACO, for object-level scene deocclusion. To train PACO, we create a large-scale dataset with 500k samples to enable self-supervised learning. Experiments on COCOA and various real-world scenes demonstrate the superior capability of PACO for scene deocclusion, surpassing the state of the arts by a large margin.
  arXiv  Detail & Related papers  (2024-06-11T20:34:10Z)
• CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion [83.30168660888913]
  We present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model.
  arXiv  Detail & Related papers  (2023-05-25T17:39:13Z)
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition [69.90530987240899]
  We present an unsupervised variational approach to this problem. Our model learns to infer two sets of latent representations from RGB video input alone. It represents object attributes in an allocentric manner which does not depend on viewpoint.
  arXiv  Detail & Related papers  (2021-06-07T17:59:23Z)
• Self-supervised Human Detection and Segmentation via Multi-view Consensus [116.92405645348185]
  We propose a multi-camera framework in which geometric constraints are embedded in the form of multi-view consistency during training. We show that our approach outperforms state-of-the-art self-supervised person detection and segmentation techniques on images that visually depart from those of standard benchmarks.
  arXiv  Detail & Related papers  (2020-12-09T15:47:21Z)
• Self-Supervised Scene De-occlusion [186.89979151728636]
  This paper investigates the problem of scene de-occlusion, which aims to recover the underlying occlusion ordering and complete the invisible parts of occluded objects. We make the first attempt to address the problem through a novel and unified framework that recovers hidden scene structures without ordering and amodal annotations as supervisions. Based on PCNet-M and PCNet-C, we devise a novel inference scheme to accomplish scene de-occlusion, via progressive ordering recovery, amodal completion and content completion.
  arXiv  Detail & Related papers  (2020-04-06T16:31:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.