CoRF : Colorizing Radiance Fields using Knowledge Distillation
- URL: http://arxiv.org/abs/2309.07668v1
- Date: Thu, 14 Sep 2023 12:30:48 GMT
- Title: CoRF : Colorizing Radiance Fields using Knowledge Distillation
- Authors: Ankit Dhiman and R Srinath and Srinjay Sarkar and Lokesh R Boregowda
and R Venkatesh Babu
- Abstract summary: This work presents a method for synthesizing colorized novel views from input grey-scale multi-view images.
We propose a distillation based method to transfer color knowledge from the colorization networks trained on natural images to the radiance field network.
The experimental results demonstrate that the proposed method produces superior colorized novel views for indoor and outdoor scenes.
- Score: 25.714166805323135
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Neural radiance field (NeRF) based methods enable high-quality novel-view
synthesis for multi-view images. This work presents a method for synthesizing
colorized novel views from input grey-scale multi-view images. When we apply
image or video-based colorization methods on the generated grey-scale novel
views, we observe artifacts due to inconsistency across views. Training a
radiance field network on the colorized grey-scale image sequence also does not
solve the 3D consistency issue. We propose a distillation based method to
transfer color knowledge from the colorization networks trained on natural
images to the radiance field network. Specifically, our method uses the
radiance field network as a 3D representation and transfers knowledge from
existing 2D colorization methods. The experimental results demonstrate that the
proposed method produces superior colorized novel views for indoor and outdoor
scenes while maintaining cross-view consistency than baselines. Further, we
show the efficacy of our method on applications like colorization of radiance
field network trained from 1.) Infra-Red (IR) multi-view images and 2.) Old
grey-scale multi-view image sequences.
Related papers
- Layered Rendering Diffusion Model for Zero-Shot Guided Image Synthesis [60.260724486834164]
This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries.
We present two key innovations: Vision Guidance and the Layered Rendering Diffusion framework.
We apply our method to three practical applications: bounding box-to-image, semantic mask-to-image and image editing.
arXiv Detail & Related papers (2023-11-30T10:36:19Z) - Novel View Synthesis from a Single RGBD Image for Indoor Scenes [4.292698270662031]
We propose an approach for synthesizing novel view images from a single RGBD (Red Green Blue-Depth) input.
In our method, we convert an RGBD image into a point cloud and render it from a different viewpoint, then formulate the NVS task into an image translation problem.
arXiv Detail & Related papers (2023-11-02T08:34:07Z) - PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields [60.66412075837952]
We present PaletteNeRF, a novel method for appearance editing of neural radiance fields (NeRF) based on 3D color decomposition.
Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases.
We extend our framework with compressed semantic features for semantic-aware appearance editing.
arXiv Detail & Related papers (2022-12-21T00:20:01Z) - Adaptive color transfer from images to terrain visualizations [0.0]
We present a two-step image-to-terrain color transfer method that can transfer color from arbitrary images to diverse terrain models.
First, we present a new image color organization method that organizes discrete, irregular image colors into a continuous, regular color grid.
We quantify a series of subjective concerns about color crafting, such as "the lower, the higher" principle, color conventions, and aerial perspectives.
arXiv Detail & Related papers (2022-05-30T08:03:30Z) - A Deep Learning Approach for Digital ColorReconstruction of Lenticular
Films [8.264186103325725]
Lenticular films emerged in the 1920s and were one of the first technologies that permitted to capture full color information in motion.
In this work, we introduce an automated, fully digital pipeline to process the scan of lenticular films and colorize the image.
Our method merges deep learning with a model-based approach in order to maximize the performance while making sure that the reconstructed colored images truthfully match the encoded color information.
arXiv Detail & Related papers (2022-02-10T11:08:50Z) - IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views.
By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
arXiv Detail & Related papers (2021-02-25T18:56:21Z) - Neural Radiance Flow for 4D View Synthesis and Video Processing [59.9116932930108]
We present a method to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images.
Key to our approach is the use of a neural implicit representation that learns to capture the 3D occupancy, radiance, and dynamics of the scene.
arXiv Detail & Related papers (2020-12-17T17:54:32Z) - Semantic View Synthesis [56.47999473206778]
We tackle a new problem of semantic view synthesis -- generating free-viewpoint rendering of a synthesized scene using a semantic label map as input.
First, we focus on synthesizing the color and depth of the visible surface of the 3D scene.
We then use the synthesized color and depth to impose explicit constraints on the multiple-plane image (MPI) representation prediction process.
arXiv Detail & Related papers (2020-08-24T17:59:46Z) - 3D Photography using Context-aware Layered Depth Inpainting [50.66235795163143]
We propose a method for converting a single RGB-D input image into a 3D photo.
A learning-based inpainting model synthesizes new local color-and-depth content into the occluded region.
The resulting 3D photos can be efficiently rendered with motion parallax.
arXiv Detail & Related papers (2020-04-09T17:59:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.