Can we make NeRF-based visual localization privacy-preserving?

• URL: http://arxiv.org/abs/2508.18971v1
• Date: Tue, 26 Aug 2025 12:17:00 GMT
• Title: Can we make NeRF-based visual localization privacy-preserving?
• Authors: Maxime Pietrantoni, Martin Humenberger, Torsten Sattler, Gabriela Csurka,
• Abstract summary: We propose a protocol to assess privacy-preservation of NeRF-based representations.<n>We show that NeRFs trained with photometric losses store fine-grained details in their geometry representations, making them vulnerable to privacy attacks.<n>We propose ppNeSF (Privacy-Preserving Neural Field), a NeRF variant trained with segmentation supervision instead of RGB images.
• Score: 30.56803525955743
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual localization (VL) is the task of estimating the camera pose in a known scene. VL methods, a.o., can be distinguished based on how they represent the scene, e.g., explicitly through a (sparse) point cloud or a collection of images or implicitly through the weights of a neural network. Recently, NeRF-based methods have become popular for VL. While NeRFs offer high-quality novel view synthesis, they inadvertently encode fine scene details, raising privacy concerns when deployed in cloud-based localization services as sensitive information could be recovered. In this paper, we tackle this challenge on two ends. We first propose a new protocol to assess privacy-preservation of NeRF-based representations. We show that NeRFs trained with photometric losses store fine-grained details in their geometry representations, making them vulnerable to privacy attacks, even if the head that predicts colors is removed. Second, we propose ppNeSF (Privacy-Preserving Neural Segmentation Field), a NeRF variant trained with segmentation supervision instead of RGB images. These segmentation labels are learned in a self-supervised manner, ensuring they are coarse enough to obscure identifiable scene details while remaining discriminativeness in 3D. The segmentation space of ppNeSF can be used for accurate visual localization, yielding state-of-the-art results.

Related papers

• Obfuscation Based Privacy Preserving Representations are Recoverable Using Neighborhood Information [43.14648227833812]
  Research on privacy-preserving localization has focused on preventing inversion attacks on query image keypoints and the 3D points of the scene map.<n>We point to a common weakness of these obfuscations that allows to recover approximations of the original point positions under the assumption of known neighborhoods.<n>Our results show that these schemes should not be considered privacy-preserving, even though they are claimed to be privacy-preserving.
  arXiv  Detail & Related papers  (2024-09-17T20:13:54Z)
• OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views [90.71215823587875]
  We propose OpenNeRF which naturally operates on posed images and directly encodes the VLM features within the NeRF. Our work shows that using pixel-wise VLM features results in an overall less complex architecture without the need for additional DINO regularization. For 3D point cloud segmentation on the Replica dataset, OpenNeRF outperforms recent open-vocabulary methods such as LERF and OpenScene by at least +4.9 mIoU.
  arXiv  Detail & Related papers  (2024-04-04T17:59:08Z)
• NeRFuser: Large-Scale Scene Representation by NeRF Fusion [35.749208740102546]
  A practical benefit of implicit visual representations like Neural Radiance Fields (NeRFs) is their memory efficiency. We propose NeRFuser, a novel architecture for NeRF registration and blending that assumes only access to pre-generated NeRFs.
  arXiv  Detail & Related papers  (2023-05-22T17:59:05Z)
• Privacy-Preserving Representations are not Enough -- Recovering Scene Content from Camera Poses [63.12979986351964]
  Existing work on privacy-preserving localization aims to defend against an attacker who has access to a cloud-based service. We show that an attacker can learn about details of a scene without any access by simply querying a localization service.
  arXiv  Detail & Related papers  (2023-05-08T10:25:09Z)
• Behind the Scenes: Density Fields for Single View Reconstruction [63.40484647325238]
  Inferring meaningful geometric scene representation from a single image is a fundamental problem in computer vision. We propose to predict implicit density fields. A density field maps every location in the frustum of the input image to volumetric density. We show that our method is able to predict meaningful geometry for regions that are occluded in the input image.
  arXiv  Detail & Related papers  (2023-01-18T17:24:01Z)
• Removing Objects From Neural Radiance Fields [60.067117643543824]
  We propose a framework to remove objects from a NeRF representation created from an RGB-D sequence. Our NeRF inpainting method leverages recent work in 2D image inpainting and is guided by a user-provided mask. We show that our method for NeRF editing is effective for synthesizing plausible inpaintings in a multi-view coherent manner.
  arXiv  Detail & Related papers  (2022-12-22T18:51:06Z)
• Privacy-Preserving Visual Localization with Event Cameras [13.21898697942957]
  Event cameras can potentially make robust localization due to high dynamic range and small motion blur. We propose applying event-to-image conversion prior to localization which leads to stable localization. In the privacy perspective, event cameras capture only a fraction of visual information compared to normal cameras.
  arXiv  Detail & Related papers  (2022-12-04T07:22:17Z)
• NeRF-SOS: Any-View Self-supervised Object Segmentation from Complex Real-World Scenes [80.59831861186227]
  This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object NeRF-SOS, encourages NeRF models to distill compact geometry-aware segmentation clusters. It consistently surpasses other 2D-based self-supervised baselines and predicts finer semantics masks than existing supervised counterparts.
  arXiv  Detail & Related papers  (2022-09-19T06:03:17Z)
• BARF: Bundle-Adjusting Neural Radiance Fields [104.97810696435766]
  We propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect camera poses. BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems.
  arXiv  Detail & Related papers  (2021-04-13T17:59:51Z)
• D-NeRF: Neural Radiance Fields for Dynamic Scenes [72.75686949608624]
  We introduce D-NeRF, a method that extends neural radiance fields to a dynamic domain. D-NeRF reconstructs images of objects under rigid and non-rigid motions from a camera moving around the scene. We demonstrate the effectiveness of our approach on scenes with objects under rigid, articulated and non-rigid motions.
  arXiv  Detail & Related papers  (2020-11-27T19:06:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.