3D Reconstruction of Protein Structures from Multi-view AFM Images using Neural Radiance Fields (NeRFs)

• URL: http://arxiv.org/abs/2408.06244v1
• Date: Mon, 12 Aug 2024 15:53:24 GMT
• Title: 3D Reconstruction of Protein Structures from Multi-view AFM Images using Neural Radiance Fields (NeRFs)
• Authors: Jaydeep Rade, Ethan Herron, Soumik Sarkar, Anwesha Sarkar, Adarsh Krishnamurthy,
• Abstract summary: We investigate using atomic force microscopy (AFM) combined with deep learning to predict the 3D structures of protein complexes (PCs) AFM generates height maps that depict the PCs in various random orientations, providing a rich information for training a neural network to predict the 3D structures. The performance of UpFusion is evaluated through zero-shot predictions of 3D protein structures using AFM images.
• Score: 8.257409049477827
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advancements in deep learning for predicting 3D protein structures have shown promise, particularly when leveraging inputs like protein sequences and Cryo-Electron microscopy (Cryo-EM) images. However, these techniques often fall short when predicting the structures of protein complexes (PCs), which involve multiple proteins. In our study, we investigate using atomic force microscopy (AFM) combined with deep learning to predict the 3D structures of PCs. AFM generates height maps that depict the PCs in various random orientations, providing a rich information for training a neural network to predict the 3D structures. We then employ the pre-trained UpFusion model (which utilizes a conditional diffusion model for synthesizing novel views) to train an instance-specific NeRF model for 3D reconstruction. The performance of UpFusion is evaluated through zero-shot predictions of 3D protein structures using AFM images. The challenge, however, lies in the time-intensive and impractical nature of collecting actual AFM images. To address this, we use a virtual AFM imaging process that transforms a `PDB' protein file into multi-view 2D virtual AFM images via volume rendering techniques. We extensively validate the UpFusion architecture using both virtual and actual multi-view AFM images. Our results include a comparison of structures predicted with varying numbers of views and different sets of views. This novel approach holds significant potential for enhancing the accuracy of protein complex structure predictions with further fine-tuning of the UpFusion network.

Related papers

• MVGamba: Unify 3D Content Generation as State Space Sequence Modeling [150.80564081817786]
  We introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor. With off-the-detail multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only $0.1times$ of the model size.
  arXiv  Detail & Related papers  (2024-06-10T15:26:48Z)
• GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement [51.97726804507328]
  We propose a novel approach for 3D mesh reconstruction from multi-view images. Our method takes inspiration from large reconstruction models that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images.
  arXiv  Detail & Related papers  (2024-06-09T05:19:24Z)
• A Generative Machine Learning Model for Material Microstructure 3D Reconstruction and Performance Evaluation [4.169915659794567]
  The dimensional extension from 2D to 3D is viewed as a highly challenging inverse problem from the current technological perspective. A novel generative model that integrates the multiscale properties of U-net with and the generative capabilities of GAN has been proposed. The model's accuracy is further improved by combining the image regularization loss with the Wasserstein distance loss.
  arXiv  Detail & Related papers  (2024-02-24T13:42:34Z)
• Multi-View Neural 3D Reconstruction of Micro-/Nanostructures with Atomic Force Microscopy [34.22380991792944]
  We propose a multi-view neural-network-based framework with Atomic Force Microscopy (MVN-AFM) MVN-AFM uses an iterative method to align multi-view data and eliminate AFM artifacts simultaneously. Experiments show that MVN-AFM effectively eliminates artifacts present in raw AFM images and reconstructs various micro-/nanostructures.
  arXiv  Detail & Related papers  (2024-01-21T16:46:04Z)
• ReconFusion: 3D Reconstruction with Diffusion Priors [104.73604630145847]
  We present ReconFusion to reconstruct real-world scenes using only a few photos. Our approach leverages a diffusion prior for novel view synthesis, trained on synthetic and multiview datasets. Our method synthesizes realistic geometry and texture in underconstrained regions while preserving the appearance of observed regions.
  arXiv  Detail & Related papers  (2023-12-05T18:59:58Z)
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction [77.69363640021503]
  3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. We present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects.
  arXiv  Detail & Related papers  (2023-04-13T17:59:01Z)
• CryoFormer: Continuous Heterogeneous Cryo-EM Reconstruction using Transformer-based Neural Representations [49.49939711956354]
  Cryo-electron microscopy (cryo-EM) allows for the high-resolution reconstruction of 3D structures of proteins and other biomolecules. It is still challenging to reconstruct the continuous motions of 3D structures from noisy and randomly oriented 2D cryo-EM images. We propose CryoFormer, a new approach for continuous heterogeneous cryo-EM reconstruction.
  arXiv  Detail & Related papers  (2023-03-28T18:59:17Z)
• 3D Reconstruction of Protein Complex Structures Using Synthesized Multi-View AFM Images [9.91587631689811]
  We train a neural network for 3D reconstruction called Pix2Vox++ using the synthesized multi-view 2D AFM images dataset. We compare the predicted structure obtained using a different number of views and get the intersection over union (IoU) value of 0.92 on the training dataset and 0.52 on the validation dataset.
  arXiv  Detail & Related papers  (2022-11-26T20:50:34Z)
• FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction [73.85709132666626]
  Existing representations, such as parametric models, voxel grids, meshes and implicit neural representations, have difficulties achieving high-quality results and real-time speed at the same time. We propose Fourier Occupancy Field (FOF), a novel powerful, efficient and flexible 3D representation, for monocular real-time and accurate human reconstruction. A FOF can be stored as a multi-channel image, which is compatible with 2D convolutional neural networks and can bridge the gap between 3D and 2D images.
  arXiv  Detail & Related papers  (2022-06-05T14:45:02Z)
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model [14.593550382914767]
  We propose NeRF-VAE, a 3D scene generative model that incorporates geometric structure via NeRF and differentiable volume rendering. NeRF-VAE's explicit 3D rendering process contrasts previous generative models with convolution-based rendering. We show that, once trained, NeRF-VAE is able to infer and render geometrically-consistent scenes from previously unseen 3D environments.
  arXiv  Detail & Related papers  (2021-04-01T16:16:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.