OpticFusion: Multi-Modal Neural Implicit 3D Reconstruction of Microstructures by Fusing White Light Interferometry and Optical Microscopy

• URL: http://arxiv.org/abs/2501.09259v1
• Date: Thu, 16 Jan 2025 03:02:08 GMT
• Title: OpticFusion: Multi-Modal Neural Implicit 3D Reconstruction of Microstructures by Fusing White Light Interferometry and Optical Microscopy
• Authors: Shuo Chen, Yijin Li, Guofeng Zhang,
• Abstract summary: White Light Interferometry (WLI) is a precise optical tool for measuring the 3D topography of microstructures.<n>Previous methods have attempted to overcome this limitation by modifying WLI hardware and analysis software.<n>We introduce OpticFusion, a novel approach that uses an additional digital optical microscope to achieve 3D reconstruction with natural color textures.
• Score: 6.928166962619509
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: White Light Interferometry (WLI) is a precise optical tool for measuring the 3D topography of microstructures. However, conventional WLI cannot capture the natural color of a sample's surface, which is essential for many microscale research applications that require both 3D geometry and color information. Previous methods have attempted to overcome this limitation by modifying WLI hardware and analysis software, but these solutions are often costly. In this work, we address this challenge from a computer vision multi-modal reconstruction perspective for the first time. We introduce OpticFusion, a novel approach that uses an additional digital optical microscope (OM) to achieve 3D reconstruction with natural color textures using multi-view WLI and OM images. Our method employs a two-step data association process to obtain the poses of WLI and OM data. By leveraging the neural implicit representation, we fuse multi-modal data and apply color decomposition technology to extract the sample's natural color. Tested on our multi-modal dataset of various microscale samples, OpticFusion achieves detailed 3D reconstructions with color textures. Our method provides an effective tool for practical applications across numerous microscale research fields. The source code and our real-world dataset are available at https://github.com/zju3dv/OpticFusion.

Related papers

• Modality-Aware Feature Matching: A Comprehensive Review of Single- and Cross-Modality Techniques [91.26187560114381]
  Feature matching is a cornerstone task in computer vision, essential for applications such as image retrieval, stereo matching, 3D reconstruction, and SLAM.<n>This survey comprehensively reviews modality-based feature matching, exploring traditional handcrafted methods and contemporary deep learning approaches.
  arXiv  Detail & Related papers  (2025-07-30T15:56:36Z)
• From Fibers to Cells: Fourier-Based Registration Enables Virtual Cresyl Violet Staining From 3D Polarized Light Imaging [32.73124984242397]
  Comprehensive assessment of the various aspects of the brain's microstructure requires the use of complementary imaging techniques.<n>The gold standard for cytoarchitectonic analysis is light microscopic imaging of cell-body stained tissue sections.<n>We take advantage of deep learning methods for image-to-image translation to generate a virtual staining of 3D-PLI that is spatially aligned at the cellular level.
  arXiv  Detail & Related papers  (2025-05-16T15:59:15Z)
• IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations [64.07859467542664]
  Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics.<n>Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs.<n>We introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.
  arXiv  Detail & Related papers  (2024-12-16T18:52:56Z)
• Textured Mesh Saliency: Bridging Geometry and Texture for Human Perception in 3D Graphics [50.23625950905638]
  We present a new dataset for textured mesh saliency, created through an innovative eye-tracking experiment in a six degrees of freedom (6-DOF) VR environment.<n>Our proposed model predicts saliency maps for textured mesh surfaces by treating each triangular face as an individual unit and assigning a saliency density value to reflect the importance of each local surface region.
  arXiv  Detail & Related papers  (2024-12-11T08:27:33Z)
• MTFusion: Reconstructing Any 3D Object from Single Image Using Multi-word Textual Inversion [10.912989885886617]
  We propose MTFusion, which leverages both image data and textual descriptions for high-fidelity 3D reconstruction. Our approach consists of two stages. First, we adopt a novel multi-word textual inversion technique to extract a detailed text description. Then, we use this description and the image to generate a 3D model with FlexiCubes.
  arXiv  Detail & Related papers  (2024-11-19T03:29:18Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• Multispectral Texture Synthesis using RGB Convolutional Neural Networks [2.3213238782019316]
  State-of-the-art RGB texture synthesis algorithms rely on style distances that are computed through statistics of deep features. We propose two solutions to extend these methods to multispectral imaging.
  arXiv  Detail & Related papers  (2024-10-21T13:49:54Z)
• Mastoidectomy Multi-View Synthesis from a Single Microscopy Image [4.777201894011511]
  We introduce a novel pipeline that is capable of generating synthetic multi-view videos from a single CI microscope image. We use a patient's pre-operative CT scan to predict the post-mastoidectomy surface using a method designed for this purpose.
  arXiv  Detail & Related papers  (2024-08-31T16:45:24Z)
• Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI [36.136619420474766]
  3D-PLI is a microscopic imaging technique that enables insights into the fine-grained organization of myelinated nerve fibers with high resolution. Best practices for observer-independent characterization of fiber architecture in 3D-PLI are not yet available. We propose the application of a fully data-driven approach to characterize nerve fiber architecture in 3D-PLI images using self-supervised representation learning.
  arXiv  Detail & Related papers  (2024-01-30T17:49:53Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo [103.08512487830669]
  We present a modern solution to the multi-view photometric stereo problem (MVPS) We procure the surface orientation using a photometric stereo (PS) image formation model and blend it with a multi-view neural radiance field representation to recover the object's surface geometry. Our method performs neural rendering of multi-view images while utilizing surface normals estimated by a deep photometric stereo network.
  arXiv  Detail & Related papers  (2021-10-11T20:20:03Z)
• Programmable 3D snapshot microscopy with Fourier convolutional networks [3.2156268397508314]
  3D snapshot microscopy enables volumetric imaging as fast as a camera allows by capturing a 3D volume in a single 2D camera image. We introduce a class of global kernel Fourier convolutional neural networks which can efficiently integrate the globally mixed information encoded in a 3D snapshot image.
  arXiv  Detail & Related papers  (2021-04-21T16:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.