Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats

• URL: http://arxiv.org/abs/2410.12781v1
• Date: Wed, 16 Oct 2024 17:54:06 GMT
• Title: Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats
• Authors: Chen Ziwen, Hao Tan, Kai Zhang, Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, Zexiang Xu,
• Abstract summary: Long-LRM can process 32 source images at 960x540 resolution within only 1.3 seconds on a single A100 80G GPU. Unlike previous feed-forward models that are limited to processing 14 input images, Long-LRM reconstructs the entire scene in a single feed-forward step.
• Score: 31.37432523412404
• License:
• Abstract: We propose Long-LRM, a generalizable 3D Gaussian reconstruction model that is capable of reconstructing a large scene from a long sequence of input images. Specifically, our model can process 32 source images at 960x540 resolution within only 1.3 seconds on a single A100 80G GPU. Our architecture features a mixture of the recent Mamba2 blocks and the classical transformer blocks which allowed many more tokens to be processed than prior work, enhanced by efficient token merging and Gaussian pruning steps that balance between quality and efficiency. Unlike previous feed-forward models that are limited to processing 1~4 input images and can only reconstruct a small portion of a large scene, Long-LRM reconstructs the entire scene in a single feed-forward step. On large-scale scene datasets such as DL3DV-140 and Tanks and Temples, our method achieves performance comparable to optimization-based approaches while being two orders of magnitude more efficient. Project page: https://arthurhero.github.io/projects/llrm

Related papers

• M-LRM: Multi-view Large Reconstruction Model [37.46572626325514]
  Multi-view Large Reconstruction Model (M-LRM) designed to efficiently reconstruct high-quality 3D shapes from multi-views in a 3D-aware manner. Compared to Large Reconstruction Model, the proposed M-LRM can produce a tri-plane NeRF with $128 times 128$ resolution and generate 3D shapes of high fidelity.
  arXiv  Detail & Related papers  (2024-06-11T18:29:13Z)
• MVGamba: Unify 3D Content Generation as State Space Sequence Modeling [150.80564081817786]
  We introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor. With off-the-detail multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only $0.1times$ of the model size.
  arXiv  Detail & Related papers  (2024-06-10T15:26:48Z)
• GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting [49.32327147931905]
  We propose GS-LRM, a scalable large reconstruction model that can predict high-quality 3D Gaussians from 2-4 posed sparse images in 0.23 seconds on single A100 GPU. Our model features a very simple transformer-based architecture; we patchify input posed images, pass the primitive multi-view image tokens through a sequence of transformer blocks, and decode final per-pixel Gaussian parameters directly from these tokens for differentiable rendering.
  arXiv  Detail & Related papers  (2024-04-30T16:47:46Z)
• Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction [153.52406455209538]
  Gamba is an end-to-end 3D reconstruction model from a single-view image. It completes reconstruction within 0.05 seconds on a single NVIDIA A100 GPU.
  arXiv  Detail & Related papers  (2024-03-27T17:40:14Z)
• GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation [85.15374487533643]
  We introduce GRM, a large-scale reconstructor capable of recovering a 3D asset from sparse-view images in around 0.1s. GRM is a feed-forward transformer-based model that efficiently incorporates multi-view information. We also showcase the potential of GRM in generative tasks, i.e., text-to-3D and image-to-3D, by integrating it with existing multi-view diffusion models.
  arXiv  Detail & Related papers  (2024-03-21T17:59:34Z)
• U-shaped Vision Mamba for Single Image Dehazing [8.134659382415185]
  We introduce Vision Mamba (UVM-Net), an efficient single-image dehazing network. Inspired by the State Space Sequence Models (SSMs), a new deep sequence model known for its power to handle long sequences, we design a Bi-SSM block. Our method takes only text0.009 seconds to infer a $325 times 325$ resolution image (100FPS) without I/O handling time.
  arXiv  Detail & Related papers  (2024-02-06T16:46:28Z)
• LRM: Large Reconstruction Model for Single Image to 3D [61.47357798633123]
  We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds. LRM adopts a highly scalable transformer-based architecture with 500 million learnable parameters to directly predict a neural radiance field (NeRF) from the input image. We train our model in an end-to-end manner on massive multi-view data containing around 1 million objects.
  arXiv  Detail & Related papers  (2023-11-08T00:03:52Z)
• Bayesian Image Reconstruction using Deep Generative Models [7.012708932320081]
  In this work, we leverage state-of-the-art (SOTA) generative models for building powerful image priors. Our method, called Bayesian Reconstruction through Generative Models (BRGM), uses a single pre-trained generator model to solve different image restoration tasks.
  arXiv  Detail & Related papers  (2020-12-08T17:11:26Z)
• Locally Masked Convolution for Autoregressive Models [107.4635841204146]
  LMConv is a simple modification to the standard 2D convolution that allows arbitrary masks to be applied to the weights at each location in the image. We learn an ensemble of distribution estimators that share parameters but differ in generation order, achieving improved performance on whole-image density estimation.
  arXiv  Detail & Related papers  (2020-06-22T17:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.