LumiTokens: 3D Relighting via Token-Space Lighting Transformation

• URL: http://arxiv.org/abs/2608.18215v1
• Date: Tue, 18 Aug 2026 18:00:21 GMT
• Title: LumiTokens: 3D Relighting via Token-Space Lighting Transformation
• Abstract summary: We present LumiTokens, a framework that formulates 3D relighting as a direct transformation on latent scene tokens.<n>Our model introduces a Scene Token Editor that processes scene tokens jointly with light-ray tokens through self-attention.<n>Experiments demonstrate that LumiTokens achieves comparable or superior relighting quality to other methods.
• Score: 19.630351342370354
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing 3D relighting methods operate through either explicit material decomposition, diffusion-based view-space generation, or a combination of both, requiring full recomputation for each new lighting condition. We observe that recent latent scene representations, which encode multi-view images into a set of compact tokens with no fixed physical semantics, open up a novel design space for relighting. We present LumiTokens, a framework that formulates 3D relighting as a direct transformation on latent scene tokens, without explicit 3D representations, rendering equations, or physics-based decomposition. Our model introduces a Scene Token Editor that processes scene tokens jointly with light-ray tokens through self-attention, producing updated tokens that can be decoded into multi-view-consistent relit images. To support diverse lighting types through a unified interface, all lighting signals, including environment maps, point lights, and area lights, are parameterized as Plucker ray tokens, enabling native 3D user interaction with a representation that carries no explicit spatial structure. Crucially, this design supports progressive relighting: because the editor's output remains in the same latent space as its input, a user can incrementally build up illumination one light source at a time, with each edit composing in token space. Experiments demonstrate that LumiTokens achieves comparable or superior relighting quality to other methods and supports progressive, composable lighting edits. Project page: https://neu-vi.github.io/LumiTokens/

Related papers

• LightFuse: Relightable Interactive Gaussian Scene Reconstruction via Multi-Scan Fusion and 2D Gaussian Ray Tracing [19.013915783052433]
  Relightable interactive scene reconstruction aims to build an editable 3D model from scans of different object arrangements and render new layouts under novel illumination.<n>We present LightFuse, a 2D Gaussian framework that extends interactive scene reconstruction with explicit material-illumination decomposition and physically based relighting.
  arXiv  Detail & Related papers  (2026-08-29T13:49:13Z)
• Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relighting [25.64036257408644]
  We propose Lume-Palette, a progressive framework for spatially controllable indoor relighting.<n>The approach decouples relighting into two stages: (1) illumination distillation, which extracts canonical illumination palettes from a pretrained diffusion model, and (2) illumination casting, which explicitly maps target spatial lighting conditions.<n>Experiments on diverse synthetic scenes and real-world scenes demonstrate that Lume-Palette produces photorealistic, spatially controllable, and multi-view consistent relighting results.
  arXiv  Detail & Related papers  (2026-07-09T19:09:48Z)
• TokenLight: Precise Lighting Control in Images using Attribute Tokens [36.19968483255676]
  We formulate relighting as a conditional image generation task and introduce attribute tokens to encode lighting factors.<n>The model is trained on a large-scale synthetic dataset with ground-truth lighting annotations.<n>We validate our approach across a variety of relighting tasks, including controlling in-scene lighting fixtures and editing environment illumination.
  arXiv  Detail & Related papers  (2026-04-16T17:59:50Z)
• SceneTok: A Compressed, Diffusable Token Space for 3D Scenes [18.094424146923892]
  We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens.<n>We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality.
  arXiv  Detail & Related papers  (2026-02-21T16:08:17Z)
• LightSwitch: Multi-view Relighting with Material-guided Diffusion [73.5965603000002]
  LightSwitch is a novel finetuned material-relighting diffusion framework.<n>We show that our 2D relighting prediction quality exceeds previous state-of-the-art relighting priors that directly relight from images.
  arXiv  Detail & Related papers  (2025-08-08T17:59:52Z)
• UVA: Towards Unified Volumetric Avatar for View Synthesis, Pose rendering, Geometry and Texture Editing [83.0396740127043]
  We propose a new approach named textbfUnified textbfVolumetric textbfAvatar (textbfUVA) that enables local editing of both geometry and texture. UVA transforms each observation point to a canonical space using a skinning motion field and represents geometry and texture in separate neural fields. Experiments on multiple human avatars demonstrate that our UVA achieves novel view synthesis and novel pose rendering.
  arXiv  Detail & Related papers  (2023-04-14T07:39:49Z)
• UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction [9.874357856580447]
  We propose a novel transformer network for Unstructured Multiple Images (UMIFormer) It exploits transformer blocks for decoupled intra-view encoding and designed blocks for token rectification. All tokens acquired from various branches are compressed into a fixed-size compact representation.
  arXiv  Detail & Related papers  (2023-02-27T17:27:45Z)
• Physically-Based Editing of Indoor Scene Lighting from a Single Image [106.60252793395104]
  We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
  arXiv  Detail & Related papers  (2022-05-19T06:44:37Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers [51.581926074686535]
  We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. The proposed TokenGAN has achieved state-of-the-art results on several widely-used image synthesis benchmarks.
  arXiv  Detail & Related papers  (2021-11-05T12:57:50Z)
• NeLF: Practical Novel View Synthesis with Neural Light Field [93.41020940730915]
  We present a practical and robust deep learning solution for the novel view synthesis of complex scenes. In our approach, a continuous scene is represented as a light field, i.e., a set of rays, each of which has a corresponding color. Our method achieves state-of-the-art novel view synthesis results while maintaining an interactive frame rate.
  arXiv  Detail & Related papers  (2021-05-15T01:20:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.