CryoFormer: Continuous Heterogeneous Cryo-EM Reconstruction using Transformer-based Neural Representations

• URL: http://arxiv.org/abs/2303.16254v3
• Date: Wed, 11 Oct 2023 06:59:47 GMT
• Title: CryoFormer: Continuous Heterogeneous Cryo-EM Reconstruction using Transformer-based Neural Representations
• Authors: Xinhang Liu, Yan Zeng, Yifan Qin, Hao Li, Jiakai Zhang, Lan Xu, Jingyi Yu
• Abstract summary: Cryo-electron microscopy (cryo-EM) allows for the high-resolution reconstruction of 3D structures of proteins and other biomolecules. It is still challenging to reconstruct the continuous motions of 3D structures from noisy and randomly oriented 2D cryo-EM images. We propose CryoFormer, a new approach for continuous heterogeneous cryo-EM reconstruction.
• Score: 49.49939711956354
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Cryo-electron microscopy (cryo-EM) allows for the high-resolution reconstruction of 3D structures of proteins and other biomolecules. Successful reconstruction of both shape and movement greatly helps understand the fundamental processes of life. However, it is still challenging to reconstruct the continuous motions of 3D structures from hundreds of thousands of noisy and randomly oriented 2D cryo-EM images. Recent advancements use Fourier domain coordinate-based neural networks to continuously model 3D conformations, yet they often struggle to capture local flexible regions accurately. We propose CryoFormer, a new approach for continuous heterogeneous cryo-EM reconstruction. Our approach leverages an implicit feature volume directly in the real domain as the 3D representation. We further introduce a novel query-based deformation transformer decoder to improve the reconstruction quality. Our approach is capable of refining pre-computed pose estimations and locating flexible regions. In experiments, our method outperforms current approaches on three public datasets (1 synthetic and 2 experimental) and a new synthetic dataset of PEDV spike protein. The code and new synthetic dataset will be released for better reproducibility of our results. Project page: https://cryoformer.github.io.

Related papers

• LaRa: Efficient Large-Baseline Radiance Fields [32.86296116177701]
  We propose a method that unifies local and global reasoning in transformer layers, resulting in improved quality and faster convergence. Our model represents scenes as Gaussian Volumes and combines this with an image encoder and Group Attention Layers for efficient feed-forward reconstruction.
  arXiv  Detail & Related papers  (2024-07-05T17:59:58Z)
• CryoSPIN: Improving Ab-Initio Cryo-EM Reconstruction with Semi-Amortized Pose Inference [30.195615398809043]
  Cryo-EM is an increasingly popular method for determining the atomic resolution 3D structure of macromolecular complexes. Recent developments in cryo-EM have focused on deep learning for which amortized inference has been used to predict pose. Here, we propose a new semi-amortized method, cryoSPIN, in which reconstruction begins with amortized inference and then switches to a form of auto-decoding.
  arXiv  Detail & Related papers  (2024-06-15T00:44:32Z)
• CryoGEM: Physics-Informed Generative Cryo-Electron Microscopy [38.57626501108458]
  We introduce physics-informed generative cryo-electron microscopy (CryoGEM) CryoGEM integrates physics-based cryo-EM simulation with a generative unpaired noise translation to generate realistic noises. Experiments show that CryoGEM is capable of generating authentic cryo-EM images.
  arXiv  Detail & Related papers  (2023-12-04T07:52:56Z)
• Improved Cryo-EM Pose Estimation and 3D Classification through Latent-Space Disentanglement [14.973360669658561]
  We propose a self-supervised variational autoencoder architecture called "HetACUMN" based on amortized inference. Results on simulated datasets show that HetACUMN generated more accurate conformational classifications than other amortized or non-amortized methods.
  arXiv  Detail & Related papers  (2023-08-09T13:41:30Z)
• NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction [64.36535692191343]
  Implicit neural representations have shown compelling results in offline 3D reconstruction and also recently demonstrated the potential for online SLAM systems. This paper addresses two key challenges: 1) seeking a criterion to measure the quality of the candidate viewpoints for the view planning based on the new representations, and 2) learning the criterion from data that can generalize to different scenes instead of hand-crafting one. Our method demonstrates significant improvements on various metrics for the rendered image quality and the geometry quality of the reconstructed 3D models when compared with variants using TSDF or reconstruction without view planning.
  arXiv  Detail & Related papers  (2022-07-22T10:05:36Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• BNV-Fusion: Dense 3D Reconstruction using Bi-level Neural Volume Fusion [85.24673400250671]
  We present Bi-level Neural Volume Fusion (BNV-Fusion), which leverages recent advances in neural implicit representations and neural rendering for dense 3D reconstruction. In order to incrementally integrate new depth maps into a global neural implicit representation, we propose a novel bi-level fusion strategy. We evaluate the proposed method on multiple datasets quantitatively and qualitatively, demonstrating a significant improvement over existing methods.
  arXiv  Detail & Related papers  (2022-04-03T19:33:09Z)
• Multi-initialization Optimization Network for Accurate 3D Human Pose and Shape Estimation [75.44912541912252]
  We propose a three-stage framework named Multi-Initialization Optimization Network (MION) In the first stage, we strategically select different coarse 3D reconstruction candidates which are compatible with the 2D keypoints of input sample. In the second stage, we design a mesh refinement transformer (MRT) to respectively refine each coarse reconstruction result via a self-attention mechanism. Finally, a Consistency Estimation Network (CEN) is proposed to find the best result from mutiple candidates by evaluating if the visual evidence in RGB image matches a given 3D reconstruction.
  arXiv  Detail & Related papers  (2021-12-24T02:43:58Z)
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction [76.1612334630256]
  We harness the power of Generative Adversarial Networks (GANs) and Deep Convolutional Neural Networks (DCNNs) to reconstruct the facial texture and shape from single images. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, facial texture reconstruction with high-frequency details.
  arXiv  Detail & Related papers  (2021-05-16T16:35:44Z)
• Learning Rotation Invariant Features for Cryogenic Electron Microscopy Image Reconstruction [1.9756768370987245]
  Cryo-Electron Microscopy (Cryo-EM) is a Nobel prize-winning technology for determining the 3D structure of particles at near-atomic resolution. Most approaches use discrete clustering which fails to capture the continuous nature of image rotation. We propose a novel method that leverages the recent development in the generative adversarial networks.
  arXiv  Detail & Related papers  (2021-01-10T13:38:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.