The Change You Want to See (Now in 3D)

• URL: http://arxiv.org/abs/2308.10417v2
• Date: Mon, 11 Sep 2023 04:03:27 GMT
• Title: The Change You Want to See (Now in 3D)
• Authors: Ragav Sachdeva, Andrew Zisserman
• Abstract summary: The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene. We contribute a change detection model that is trained entirely on synthetic data and is class-agnostic. We release a new evaluation dataset consisting of real-world image pairs with human-annotated differences.
• Score: 65.61789642291636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene acquired from different camera positions and at different temporal instances. The open-set nature of this problem, occlusions/dis-occlusions due to the shift in viewpoint, and the lack of suitable training datasets, presents substantial challenges in devising a solution. To address this problem, we contribute a change detection model that is trained entirely on synthetic data and is class-agnostic, yet it is performant out-of-the-box on real world images without requiring fine-tuning. Our solution entails a "register and difference" approach that leverages self-supervised frozen embeddings and feature differences, which allows the model to generalise to a wide variety of scenes and domains. The model is able to operate directly on two RGB images, without requiring access to ground truth camera intrinsics, extrinsics, depth maps, point clouds, or additional before-after images. Finally, we collect and release a new evaluation dataset consisting of real-world image pairs with human-annotated differences and demonstrate the efficacy of our method. The code, datasets and pre-trained model can be found at: https://github.com/ragavsachdeva/CYWS-3D

Related papers

• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
  DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
  arXiv  Detail & Related papers  (2024-06-17T21:15:13Z)
• Inverse Neural Rendering for Explainable Multi-Object Tracking [35.072142773300655]
  We recast 3D multi-object tracking from RGB cameras as an emphInverse Rendering (IR) problem. We optimize an image loss over generative latent spaces that inherently disentangle shape and appearance properties. We validate the generalization and scaling capabilities of our method by learning the generative prior exclusively from synthetic data.
  arXiv  Detail & Related papers  (2024-04-18T17:37:53Z)
• VaLID: Variable-Length Input Diffusion for Novel View Synthesis [36.57742242154048]
  Novel View Synthesis (NVS), which tries to produce a realistic image at the target view given source view images and their corresponding poses, is a fundamental problem in 3D Vision. We try to process each pose image pair separately and then fuse them as a unified visual representation which will be injected into the model. The Multi-view Cross Former module is proposed which maps variable-length input data to fix-size output data.
  arXiv  Detail & Related papers  (2023-12-14T12:52:53Z)
• Towards Generalizable Multi-Camera 3D Object Detection via Perspective Debiasing [28.874014617259935]
  Multi-Camera 3D Object Detection (MC3D-Det) has gained prominence with the advent of bird's-eye view (BEV) approaches. We propose a novel method that aligns 3D detection with 2D camera plane results, ensuring consistent and accurate detections.
  arXiv  Detail & Related papers  (2023-10-17T15:31:28Z)
• Learning 3D Photography Videos via Self-supervised Diffusion on Single Images [105.81348348510551]
  3D photography renders a static image into a video with appealing 3D visual effects. Existing approaches typically first conduct monocular depth estimation, then render the input frame to subsequent frames with various viewpoints. We present a novel task: out-animation, which extends the space and time of input objects.
  arXiv  Detail & Related papers  (2023-02-21T16:18:40Z)
• Self-supervised Wide Baseline Visual Servoing via 3D Equivariance [35.93323183558956]
  This paper presents a novel self-supervised visual servoing method for wide baseline images. Existing approaches that regress absolute camera pose with respect to an object require 3D ground truth data of the object. Ours yields more than 35% average distance error reduction and more than 90% success rate with 3cm error tolerance.
  arXiv  Detail & Related papers  (2022-09-12T17:38:26Z)
• CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields [67.76151996543588]
  We learn a 3D- and camera-aware generative model which faithfully recovers not only the image but also the camera data distribution. At test time, our model generates images with explicit control over the camera as well as the shape and appearance of the scene.
  arXiv  Detail & Related papers  (2021-03-31T17:59:24Z)
• Deep Bingham Networks: Dealing with Uncertainty and Ambiguity in Pose Estimation [74.76155168705975]
  Deep Bingham Networks (DBN) can handle pose-related uncertainties and ambiguities arising in almost all real life applications concerning 3D data. DBN extends the state of the art direct pose regression networks by (i) a multi-hypotheses prediction head which can yield different distribution modes. We propose new training strategies so as to avoid mode or posterior collapse during training and to improve numerical stability.
  arXiv  Detail & Related papers  (2020-12-20T19:20:26Z)
• Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis [72.34794624243281]
  We propose a self-supervised learning framework to disentangle variations from unlabeled video frames. Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
  arXiv  Detail & Related papers  (2020-04-09T07:55:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.