From Diffusion to Resolution: Leveraging 2D Diffusion Models for 3D Super-Resolution Task

• URL: http://arxiv.org/abs/2411.16792v1
• Date: Mon, 25 Nov 2024 09:12:55 GMT
• Title: From Diffusion to Resolution: Leveraging 2D Diffusion Models for 3D Super-Resolution Task
• Authors: Bohao Chen, Yanchao Zhang, Yanan Lv, Hua Han, Xi Chen,
• Abstract summary: We present a novel approach that leverages the 2D diffusion model and lateral continuity within the volume to enhance 3D volume electron microscopy (vEM) super-resolution. Our results on two publicly available focused ion beam scanning electron microscopy (FIB-SEM) datasets demonstrate the robustness and practical applicability of our framework.
• Score: 19.56372155146739
• License:
• Abstract: Diffusion models have recently emerged as a powerful technique in image generation, especially for image super-resolution tasks. While 2D diffusion models significantly enhance the resolution of individual images, existing diffusion-based methods for 3D volume super-resolution often struggle with structure discontinuities in axial direction and high sampling costs. In this work, we present a novel approach that leverages the 2D diffusion model and lateral continuity within the volume to enhance 3D volume electron microscopy (vEM) super-resolution. We first simulate lateral degradation with slices in the XY plane and train a 2D diffusion model to learn how to restore the degraded slices. The model is then applied slice-by-slice in the lateral direction of low-resolution volume, recovering slices while preserving inherent lateral continuity. Following this, a high-frequency-aware 3D super-resolution network is trained on the recovery lateral slice sequences to learn spatial feature transformation across slices. Finally, the network is applied to infer high-resolution volumes in the axial direction, enabling 3D super-resolution. We validate our approach through comprehensive evaluations, including image similarity assessments, resolution analysis, and performance on downstream tasks. Our results on two publicly available focused ion beam scanning electron microscopy (FIB-SEM) datasets demonstrate the robustness and practical applicability of our framework for 3D volume super-resolution.

Related papers

• Reference-free Axial Super-resolution of 3D Microscopy Images using Implicit Neural Representation with a 2D Diffusion Prior [4.1326413814647545]
  Training a learning-based 3D super-resolution model requires ground truth isotropic volumes and suffers from the curse of dimensionality. Existing methods utilize 2D neural networks to reconstruct each axial slice, eventually piecing together the entire volume. We present a reconstruction framework based on implicit neural representation (INR), which allows 3D coherency even when optimized by independent axial slices.
  arXiv  Detail & Related papers  (2024-08-16T09:14:12Z)
• GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision [49.839374549646884]
  This paper presents GEOcc, a Geometric-Enhanced Occupancy network tailored for vision-only surround-view perception. Our approach achieves State-Of-The-Art performance on the Occ3D-nuScenes dataset with the least image resolution needed and the most weightless image backbone.
  arXiv  Detail & Related papers  (2024-05-17T07:31:20Z)
• SC-Diff: 3D Shape Completion with Latent Diffusion Models [4.913210912019975]
  This paper introduces a 3D shape completion approach using a 3D latent diffusion model optimized for completing shapes. Our method combines image-based conditioning through cross-attention and spatial conditioning through the integration of 3D features from captured partial scans.
  arXiv  Detail & Related papers  (2024-03-19T06:01:11Z)
• What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs [82.3936309001633]
  3D-aware Generative Adversarial Networks (GANs) have shown remarkable progress in learning to generate multi-view-consistent images and 3D geometries. Yet, the significant memory and computational costs of dense sampling in volume rendering have forced 3D GANs to adopt patch-based training or employ low-resolution rendering with post-processing 2D super resolution. We propose techniques to scale neural volume rendering to the much higher resolution of native 2D images, thereby resolving fine-grained 3D geometry with unprecedented detail.
  arXiv  Detail & Related papers  (2024-01-04T18:50:38Z)
• StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D [88.66678730537777]
  We present StableDreamer, a methodology incorporating three advances. First, we formalize the equivalence of the SDS generative prior and a simple supervised L2 reconstruction loss. Second, our analysis shows that while image-space diffusion contributes to geometric precision, latent-space diffusion is crucial for vivid color rendition.
  arXiv  Detail & Related papers  (2023-12-02T02:27:58Z)
• Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models [52.529394863331326]
  We propose a novel approach using two perpendicular pre-trained 2D diffusion models to solve the 3D inverse problem. Our method is highly effective for 3D medical image reconstruction tasks, including MRI Z-axis super-resolution, compressed sensing MRI, and sparse-view CT.
  arXiv  Detail & Related papers  (2023-03-15T08:28:06Z)
• Neural Volume Super-Resolution [49.879789224455436]
  We propose a neural super-resolution network that operates directly on the volumetric representation of the scene. To realize our method, we devise a novel 3D representation that hinges on multiple 2D feature planes. We validate the proposed method by super-resolving multi-view consistent views on a diverse set of unseen 3D scenes.
  arXiv  Detail & Related papers  (2022-12-09T04:54:13Z)
• Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models [33.343489006271255]
  Diffusion models have emerged as the new state-of-the-art generative model with high quality samples. We propose to augment the 2D diffusion prior with a model-based prior in the remaining direction at test time, such that one can achieve coherent reconstructions across all dimensions. Our method can be run in a single commodity GPU, and establishes the new state-of-the-art.
  arXiv  Detail & Related papers  (2022-11-19T10:32:21Z)
• Revisiting 3D Context Modeling with Supervised Pre-training for Universal Lesion Detection in CT Slices [48.85784310158493]
  We propose a Modified Pseudo-3D Feature Pyramid Network (MP3D FPN) to efficiently extract 3D context enhanced 2D features for universal lesion detection in CT slices. With the novel pre-training method, the proposed MP3D FPN achieves state-of-the-art detection performance on the DeepLesion dataset. The proposed 3D pre-trained weights can potentially be used to boost the performance of other 3D medical image analysis tasks.
  arXiv  Detail & Related papers  (2020-12-16T07:11:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.