SimVS: Simulating World Inconsistencies for Robust View Synthesis

• URL: http://arxiv.org/abs/2412.07696v1
• Date: Tue, 10 Dec 2024 17:35:12 GMT
• Title: SimVS: Simulating World Inconsistencies for Robust View Synthesis
• Authors: Alex Trevithick, Roni Paiss, Philipp Henzler, Dor Verbin, Rundi Wu, Hadi Alzayer, Ruiqi Gao, Ben Poole, Jonathan T. Barron, Aleksander Holynski, Ravi Ramamoorthi, Pratul P. Srinivasan,
• Abstract summary: We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations.
• Score: 102.83898965828621
• License:
• Abstract: Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: https://alextrevithick.github.io/simvs

Related papers

• CFSynthesis: Controllable and Free-view 3D Human Video Synthesis [57.561237409603066]
  CFSynthesis is a novel framework for generating high-quality human videos with customizable attributes. Our method leverages a texture-SMPL-based representation to ensure consistent and stable character appearances across free viewpoints. Results on multiple datasets show that CFSynthesis achieves state-of-the-art performance in complex human animations.
  arXiv  Detail & Related papers  (2024-12-15T05:57:36Z)
• View-Invariant Policy Learning via Zero-Shot Novel View Synthesis [26.231630397802785]
  We investigate how knowledge from large-scale visual data of the world may be used to address one axis of variation for generalizable manipulation: observational viewpoint. We study single-image novel view synthesis models, which learn 3D-aware scene-level priors by rendering images of the same scene from alternate camera viewpoints. For practical application to diverse robotic data, these models must operate zero-shot, performing view synthesis on unseen tasks and environments.
  arXiv  Detail & Related papers  (2024-09-05T16:39:21Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• Modeling Ambient Scene Dynamics for Free-view Synthesis [31.233859111566613]
  We introduce a novel method for dynamic free-view synthesis of an ambient scenes from a monocular capture. Our method builds upon the recent advancements in 3D Gaussian Splatting (3DGS) that can faithfully reconstruct complex static scenes.
  arXiv  Detail & Related papers  (2024-06-13T17:59:11Z)
• Towards 3D Scene Understanding by Referring Synthetic Models [65.74211112607315]
  Methods typically alleviate on-extensive annotations on real scene scans. We explore how synthetic models rely on real scene categories of synthetic features to a unified feature space. Experiments show that our method achieves the average mAP of 46.08% on the ScanNet S3DIS dataset and 55.49% by learning datasets.
  arXiv  Detail & Related papers  (2022-03-20T13:06:15Z)
• Deep 3D Mask Volume for View Synthesis of Dynamic Scenes [49.45028543279115]
  We introduce a multi-view video dataset, captured with a custom 10-camera rig in 120FPS. The dataset contains 96 high-quality scenes showing various visual effects and human interactions in outdoor scenes. We develop a new algorithm, Deep 3D Mask Volume, which enables temporally-stable view extrapolation from binocular videos of dynamic scenes, captured by static cameras.
  arXiv  Detail & Related papers  (2021-08-30T17:55:28Z)
• PixelSynth: Generating a 3D-Consistent Experience from a Single Image [30.64117903216323]
  We present an approach that fuses 3D reasoning with autoregressive modeling to outpaint large view changes in a 3D-consistent manner. We demonstrate considerable improvement in single image large-angle view synthesis results compared to a variety of methods and possible variants.
  arXiv  Detail & Related papers  (2021-08-12T17:59:31Z)
• Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data [0.059110875077162096]
  We propose a novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes. By introducing global learnable textures and a lighting-invariant view-consistency loss, our method produces consistent translations of arbitrary views. By extending existing image-based methods to view-consistent videos, we aim to impact the applicability of simulated training and evaluation environments for surgical applications.
  arXiv  Detail & Related papers  (2021-03-31T16:31:26Z)
• Future Urban Scenes Generation Through Vehicles Synthesis [90.1731992199415]
  We propose a deep learning pipeline to predict the visual future appearance of an urban scene. We follow a two stages approach, where interpretable information is included in the loop and each actor is modelled independently. We show the superiority of this approach over traditional end-to-end scene-generation methods on CityFlow.
  arXiv  Detail & Related papers  (2020-07-01T08:40:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.