Puzzle Similarity: A Perceptually-guided Cross-Reference Metric for Artifact Detection in 3D Scene Reconstructions

• URL: http://arxiv.org/abs/2411.17489v3
• Date: Tue, 29 Jul 2025 14:56:41 GMT
• Title: Puzzle Similarity: A Perceptually-guided Cross-Reference Metric for Artifact Detection in 3D Scene Reconstructions
• Abstract summary: We propose a new cross-reference metric, Puzzle Similarity, to localize artifacts in novel views.<n>Our method achieves state-of-the-art localization of artifacts in novel views, correlating with human assessment.<n>We can leverage our new metric to enhance applications like automatic image restoration, guided acquisition, or 3D reconstruction from sparse inputs.
• Score: 6.735527828654709
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern reconstruction techniques can effectively model complex 3D scenes from sparse 2D views. However, automatically assessing the quality of novel views and identifying artifacts is challenging due to the lack of ground truth images and the limitations of no-reference image metrics in predicting reliable artifact maps. The absence of such metrics hinders assessment of the quality of novel views and limits the adoption of post-processing techniques, such as inpainting, to enhance reconstruction quality. To tackle this, recent work has established a new category of metrics (cross-reference), predicting image quality solely by leveraging context from alternate viewpoint captures (arXiv:2404.14409). In this work, we propose a new cross-reference metric, Puzzle Similarity, which is designed to localize artifacts in novel views. Our approach utilizes image patch statistics from the training views to establish a scene-specific distribution, later used to identify poorly reconstructed regions in the novel views. Given the lack of good measures to evaluate cross-reference methods in the context of 3D reconstruction, we collected a novel human-labeled dataset of artifact and distortion maps in unseen reconstructed views. Through this dataset, we demonstrate that our method achieves state-of-the-art localization of artifacts in novel views, correlating with human assessment, even without aligned references. We can leverage our new metric to enhance applications like automatic image restoration, guided acquisition, or 3D reconstruction from sparse inputs. Find the project page at https://nihermann.github.io/puzzlesim/ .

Related papers

• GeoQuery: Geometry-Query Diffusion for Sparse-View Reconstruction [28.535167559081852]
  3D Gaussian Splatting (3DGS) has emerged as a prominent paradigm for 3D reconstruction and novel view synthesis.<n>We propose GeoQuery, a geometry-guided diffusion framework that integrates generative priors with explicit geometric cues.<n>GeoQuery can be seamlessly integrated into existing diffusion-based pipelines, enabling robust reconstruction even under extreme view sparsity.
  arXiv  Detail & Related papers  (2026-05-12T17:00:47Z)
• Robust Visual Localization in Compute-Constrained Environments by Salient Edge Rendering and Weighted Hamming Similarity [1.2621048930928775]
  We consider the problem of vision-based 6-DoF object pose estimation in the context of the notional Mars Sample Return campaign.<n>We propose a novel localization algorithm leveraging a custom metric together with a new template matching to the edge domain.<n>Our method consistently beats the state of the art in compute and memory-constrained localization, both in terms of robustness and accuracy, in turn enabling new possibilities for cheap and reliable localization on general-purpose hardware.
  arXiv  Detail & Related papers  (2025-09-29T21:22:35Z)
• SparseRecon: Neural Implicit Surface Reconstruction from Sparse Views with Feature and Depth Consistencies [48.99420012507374]
  We propose SparseRecon, a novel neural implicit reconstruction method for sparse views with volume rendering-based feature consistency and uncertainty-guided depth constraint.<n>We show that our method outperforms the state-of-the-art methods, which can produce high-quality geometry with sparse-view input.
  arXiv  Detail & Related papers  (2025-08-01T06:51:32Z)
• Mismatched: Evaluating the Limits of Image Matching Approaches and Benchmarks [9.388897214344572]
  Three-dimensional (3D) reconstruction from two-dimensional images is an active research field in computer vision. Traditionally, parametric techniques have been employed for this task. Recent advancements have seen a shift towards learning-based methods.
  arXiv  Detail & Related papers  (2024-08-29T11:16:34Z)
• MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References [49.71130133080821]
  MaRINeR is a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations.
  arXiv  Detail & Related papers  (2024-07-18T17:50:03Z)
• Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering [93.94371335579321]
  We propose a learning framework that trains models to predict geometry-preserving depth without requiring extra data or annotations. Comprehensive experiments underscore our framework's superior generalization capabilities. Our innovative loss functions empower the model to autonomously recover domain-specific scale-and-shift coefficients.
  arXiv  Detail & Related papers  (2023-09-18T12:36:39Z)
• Single-view 3D Mesh Reconstruction for Seen and Unseen Categories [69.29406107513621]
  Single-view 3D Mesh Reconstruction is a fundamental computer vision task that aims at recovering 3D shapes from single-view RGB images. This paper tackles Single-view 3D Mesh Reconstruction, to study the model generalization on unseen categories. We propose an end-to-end two-stage network, GenMesh, to break the category boundaries in reconstruction.
  arXiv  Detail & Related papers  (2022-08-04T14:13:35Z)
• NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction [64.36535692191343]
  Implicit neural representations have shown compelling results in offline 3D reconstruction and also recently demonstrated the potential for online SLAM systems. This paper addresses two key challenges: 1) seeking a criterion to measure the quality of the candidate viewpoints for the view planning based on the new representations, and 2) learning the criterion from data that can generalize to different scenes instead of hand-crafting one. Our method demonstrates significant improvements on various metrics for the rendered image quality and the geometry quality of the reconstructed 3D models when compared with variants using TSDF or reconstruction without view planning.
  arXiv  Detail & Related papers  (2022-07-22T10:05:36Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Black-Box Test-Time Shape REFINEment for Single View 3D Reconstruction [57.805334118057665]
  We propose REFINE, a postprocessing mesh refinement step that can be easily integrated into the pipeline of any black-box method in the literature. At test time, REFINE optimize a network per mesh instance, to encourage consistency between the mesh and the given object view.
  arXiv  Detail & Related papers  (2021-08-23T03:28:47Z)
• Unsupervised Learning of 3D Object Categories from Videos in the Wild [75.09720013151247]
  We focus on learning a model from multiple views of a large collection of object instances. We propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T17:57:01Z)
• Next-best-view Regression using a 3D Convolutional Neural Network [0.9449650062296823]
  We propose a data-driven approach to address the next-best-view problem. The proposed approach trains a 3D convolutional neural network with previous reconstructions in order to regress the btxtposition of the next-best-view. We have validated the proposed approach making use of two groups of experiments.
  arXiv  Detail & Related papers  (2021-01-23T01:50:26Z)
• Single View Metrology in the Wild [94.7005246862618]
  We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications including virtual object insertion.
  arXiv  Detail & Related papers  (2020-07-18T22:31:33Z)
• Novel Object Viewpoint Estimation through Reconstruction Alignment [45.16865218423492]
  We learn a reconstruct and align approach to estimate the viewpoint of a novel object. In particular, we propose learning two networks: the first maps images to a 3D geometry-aware feature bottleneck and is trained via an image-to-image translation loss. At test time, our model finds the relative transformation that best aligns the bottleneck features of our test image to a reference image.
  arXiv  Detail & Related papers  (2020-06-05T17:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.