LiTo: Surface Light Field Tokenization

• URL: http://arxiv.org/abs/2603.11047v1
• Date: Wed, 11 Mar 2026 17:59:59 GMT
• Title: LiTo: Surface Light Field Tokenization
• Abstract summary: We propose a 3D latent representation that jointly models object geometry and view-dependent appearance.<n>Our approach achieves higher visual quality and better input fidelity than existing methods.
• Score: 27.584805504757824
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We propose a 3D latent representation that jointly models object geometry and view-dependent appearance. Most prior works focus on either reconstructing 3D geometry or predicting view-independent diffuse appearance, and thus struggle to capture realistic view-dependent effects. Our approach leverages that RGB-depth images provide samples of a surface light field. By encoding random subsamples of this surface light field into a compact set of latent vectors, our model learns to represent both geometry and appearance within a unified 3D latent space. This representation reproduces view-dependent effects such as specular highlights and Fresnel reflections under complex lighting. We further train a latent flow matching model on this representation to learn its distribution conditioned on a single input image, enabling the generation of 3D objects with appearances consistent with the lighting and materials in the input. Experiments show that our approach achieves higher visual quality and better input fidelity than existing methods.

Related papers

• Neural Acquisition & Representation of Subsurface Scattering [6.620925366901447]
  We present a method to acquire and estimate the sub-surface scattering properties of light transport at a highly detailed level by learning the pixel footprint response at each point on the object surface.<n>A stereo projector-camera setup using phase-shifted profilometry patterns efficiently captures the data for a variety of scattering objects.
  arXiv  Detail & Related papers  (2026-06-01T14:14:38Z)
• Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors [61.34273238077091]
  We present a novel method for generating orbital videos from a single image of an object.<n>Our method achieves superior visual quality, shape realism and multi-view consistency compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-14T05:35:46Z)
• 3DPR: Single Image 3D Portrait Relight using Generative Priors [101.74130664920868]
  3DPR is an image-based relighting model that leverages generative priors learnt from multi-view One-Light-at-A-Time (OLAT) images.<n>We leverage the latent space of a pre-trained generative head model that provides a rich prior over face geometry learnt from in-the-wild image datasets.<n>Our reflectance network operates in the latent space of the generative head model, crucially enabling a relatively small number of lightstage images to train the reflectance model.
  arXiv  Detail & Related papers  (2025-10-17T17:37:42Z)
• BiGS: Bidirectional Gaussian Primitives for Relightable 3D Gaussian Splatting [10.918133974256913]
  We present Bidirectional Gaussian Primitives, an image-based novel view synthesis technique. Our approach integrates light intrinsic decomposition into the Gaussian splatting framework, enabling real-time relighting of 3D objects.
  arXiv  Detail & Related papers  (2024-08-23T21:04:40Z)
• PBIR-NIE: Glossy Object Capture under Non-Distant Lighting [30.325872237020395]
  Glossy objects present a significant challenge for 3D reconstruction from multi-view input images under natural lighting. We introduce PBIR-NIE, an inverse rendering framework designed to holistically capture the geometry, material attributes, and surrounding illumination of such objects.
  arXiv  Detail & Related papers  (2024-08-13T13:26:24Z)
• PT43D: A Probabilistic Transformer for Generating 3D Shapes from Single Highly-Ambiguous RGB Images [26.900974153235456]
  We propose a transformer-based autoregressive model to generate 3D shapes from single RGB images. To handle realistic scenarios such as field-of-view truncation, we create simulated image-to-shape training pairs. We then adopt cross-attention to effectively identify the most relevant region of interest from the input image for shape generation.
  arXiv  Detail & Related papers  (2024-05-20T09:49:13Z)
• Relightify: Relightable 3D Faces from a Single Image via Diffusion Models [86.3927548091627]
  We present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent estimation.
  arXiv  Detail & Related papers  (2023-05-10T11:57:49Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes [75.63464905190061]
  Complex light paths induced by refraction and reflection have prevented both traditional and deep multiview stereo from solving this problem. We propose a physically-based network to recover 3D shape of transparent objects using a few images acquired with a mobile phone camera. Our experiments show successful recovery of high-quality 3D geometry for complex transparent shapes using as few as 5-12 natural images.
  arXiv  Detail & Related papers  (2020-04-22T23:51:30Z)
• Learning Implicit Surface Light Fields [34.89812112073539]
  Implicit representations of 3D objects have recently achieved impressive results on learning-based 3D reconstruction tasks. We propose a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field. Our model is able to infer rich visual appearance including shadows and specular reflections.
  arXiv  Detail & Related papers  (2020-03-27T13:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.