ChromaDistill: Colorizing Monochrome Radiance Fields with Knowledge Distillation

• URL: http://arxiv.org/abs/2309.07668v2
• Date: Fri, 06 Dec 2024 07:11:33 GMT
• Title: ChromaDistill: Colorizing Monochrome Radiance Fields with Knowledge Distillation
• Authors: Ankit Dhiman, R Srinath, Srinjay Sarkar, Lokesh R Boregowda, R Venkatesh Babu,
• Abstract summary: We present a method for colorized novel views from input grayscale multi-view images.<n>We propose a distillation-based method that transfers color from these networks trained on natural images to the target 3D representation.<n>Our method is agnostic to the underlying 3D representation and easily generalizable to NeRF and 3DGS methods.
• Score: 23.968181738235266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Colorization is a well-explored problem in the domains of image and video processing. However, extending colorization to 3D scenes presents significant challenges. Recent Neural Radiance Field (NeRF) and Gaussian-Splatting(3DGS) methods enable high-quality novel-view synthesis for multi-view images. However, the question arises: How can we colorize these 3D representations? This work presents a method for synthesizing colorized novel views from input grayscale multi-view images. Using image or video colorization methods to colorize novel views from these 3D representations naively will yield output with severe inconsistencies. We introduce a novel method to use powerful image colorization models for colorizing 3D representations. We propose a distillation-based method that transfers color from these networks trained on natural images to the target 3D representation. Notably, this strategy does not add any additional weights or computational overhead to the original representation during inference. Extensive experiments demonstrate that our method produces high-quality colorized views for indoor and outdoor scenes, showcasing significant cross-view consistency advantages over baseline approaches. Our method is agnostic to the underlying 3D representation and easily generalizable to NeRF and 3DGS methods. Further, we validate the efficacy of our approach in several diverse applications: 1.) Infra-Red (IR) multi-view images and 2.) Legacy grayscale multi-view image sequences. Project Webpage: https://val.cds.iisc.ac.in/chroma-distill.github.io/

Related papers

• Geometry-Aware Diffusion Models for Multiview Scene Inpainting [24.963896970130065]
  We focus on 3D scene inpainting, where parts of an input image set, captured from different viewpoints, are masked out. Most recent work addresses this challenge by combining generative models with a 3D radiance field to fuse information across viewpoints. We introduce a geometry-aware conditional generative model, capable of multi-view consistent inpainting.
  arXiv  Detail & Related papers  (2025-02-18T23:30:10Z)
• A Nerf-Based Color Consistency Method for Remote Sensing Images [0.5735035463793009]
  We propose a NeRF-based method of color consistency for multi-view images, which weaves image features together using implicit expressions, and then re-illuminates feature space to generate a fusion image with a new perspective. Experimental results show that the synthesize image generated by our method has excellent visual effect and smooth color transition at the edges.
  arXiv  Detail & Related papers  (2024-11-08T13:26:07Z)
• Transforming Color: A Novel Image Colorization Method [8.041659727964305]
  This paper introduces a novel method for image colorization that utilizes a color transformer and generative adversarial networks (GANs) The proposed method integrates a transformer architecture to capture global information and a GAN framework to improve visual quality. Experimental results show that the proposed network significantly outperforms other state-of-the-art colorization techniques.
  arXiv  Detail & Related papers  (2024-10-07T07:23:42Z)
• MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation [54.27399121779011]
  We present MVD-Fusion: a method for single-view 3D inference via generative modeling of multi-view-consistent RGB-D images. We show that our approach can yield more accurate synthesis compared to recent state-of-the-art, including distillation-based 3D inference and prior multi-view generation methods.
  arXiv  Detail & Related papers  (2024-04-04T17:59:57Z)
• Layered Rendering Diffusion Model for Zero-Shot Guided Image Synthesis [60.260724486834164]
  This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries. We present two key innovations: Vision Guidance and the Layered Rendering Diffusion framework. We apply our method to three practical applications: bounding box-to-image, semantic mask-to-image and image editing.
  arXiv  Detail & Related papers  (2023-11-30T10:36:19Z)
• ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image [15.997195076224312]
  We present a novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure. We show that our 3D reconstructions remain more faithful to the input and produce more consistent 3D models.
  arXiv  Detail & Related papers  (2023-11-09T09:17:10Z)
• Novel View Synthesis from a Single RGBD Image for Indoor Scenes [4.292698270662031]
  We propose an approach for synthesizing novel view images from a single RGBD (Red Green Blue-Depth) input. In our method, we convert an RGBD image into a point cloud and render it from a different viewpoint, then formulate the NVS task into an image translation problem.
  arXiv  Detail & Related papers  (2023-11-02T08:34:07Z)
• PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields [60.66412075837952]
  We present PaletteNeRF, a novel method for appearance editing of neural radiance fields (NeRF) based on 3D color decomposition. Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases. We extend our framework with compressed semantic features for semantic-aware appearance editing.
  arXiv  Detail & Related papers  (2022-12-21T00:20:01Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Adaptive color transfer from images to terrain visualizations [0.0]
  We present a two-step image-to-terrain color transfer method that can transfer color from arbitrary images to diverse terrain models. First, we present a new image color organization method that organizes discrete, irregular image colors into a continuous, regular color grid. We quantify a series of subjective concerns about color crafting, such as "the lower, the higher" principle, color conventions, and aerial perspectives.
  arXiv  Detail & Related papers  (2022-05-30T08:03:30Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
  We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
  arXiv  Detail & Related papers  (2021-02-25T18:56:21Z)
• Neural Radiance Flow for 4D View Synthesis and Video Processing [59.9116932930108]
  We present a method to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images. Key to our approach is the use of a neural implicit representation that learns to capture the 3D occupancy, radiance, and dynamics of the scene.
  arXiv  Detail & Related papers  (2020-12-17T17:54:32Z)
• Semantic View Synthesis [56.47999473206778]
  We tackle a new problem of semantic view synthesis -- generating free-viewpoint rendering of a synthesized scene using a semantic label map as input. First, we focus on synthesizing the color and depth of the visible surface of the 3D scene. We then use the synthesized color and depth to impose explicit constraints on the multiple-plane image (MPI) representation prediction process.
  arXiv  Detail & Related papers  (2020-08-24T17:59:46Z)
• Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild [96.09941542587865]
  We present a novel 3D pose refinement approach based on differentiable rendering for objects of arbitrary categories in the wild. In this way, we precisely align 3D models to objects in RGB images which results in significantly improved 3D pose estimates. We evaluate our approach on the challenging Pix3D dataset and achieve up to 55% relative improvement compared to state-of-the-art refinement methods in multiple metrics.
  arXiv  Detail & Related papers  (2020-07-17T12:34:38Z)
• 3D Photography using Context-aware Layered Depth Inpainting [50.66235795163143]
  We propose a method for converting a single RGB-D input image into a 3D photo. A learning-based inpainting model synthesizes new local color-and-depth content into the occluded region. The resulting 3D photos can be efficiently rendered with motion parallax.
  arXiv  Detail & Related papers  (2020-04-09T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.