Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimation

• URL: http://arxiv.org/abs/2605.03359v1
• Date: Tue, 05 May 2026 04:37:00 GMT
• Title: Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimation
• Abstract summary: Mix3R is a novel generative 3D reconstruction method that mixes feed-forward reconstruction and 3D generation.<n>Our method produces 3D shapes with better input alignment compared with pure 3D generative methods.
• Score: 43.71273997834654
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent trends in sparse-view 3D reconstruction have taken two different paths: feed-forward reconstruction that predicts pixel-aligned point maps without a complete geometry, and generative 3D reconstruction that generates complete geometry but often with poor input-alignment. We present Mix3R, a novel generative 3D reconstruction method which mixes feed-forward reconstruction and 3D generation into a single framework in an aligned manner. Mix3R generates a 3D shape in two stages: a sparse voxel generation stage and a textured geometry generation stage. Unlike pure generative methods, our first-stage generation jointly produces a coarse 3D structure (sparse voxels), per-view point maps and camera parameters aligned to that 3D structure. This is made possible by introducing a Mixture-of-Transformers architecture that inserts global self-attentions to a feed-forward reconstruction model and a 3D generative model, both pretrained on large-scale data. This design effectively retains the pretrained priors but enables better 2D-3D alignment. Based on the initial aligned generations of sparse 3D voxels and point maps, we compute an overlap-based attention bias that is directly added to another pretrained textured geometry generation model, enabling it to correctly place input textures onto generated shapes in a training-free manner. Our design brings mutual benefits to both feed-forward reconstruction and 3D generation: The feed-forward branch learns to ground its predictions to a generative 3D prior, and conversely, the 3D generation branch is conditioned on geometrically informative features from the feed-forward branch. As a result, our method produces 3D shapes with better input alignment compared with pure 3D generative methods, together with camera pose estimations more accurate than previous feed-forward reconstruction methods. Our project page is at https://jsnln.github.io/mix3r/

Related papers

• Pano3D: Unified 3D Reconstruction and Panoptic Segmentation [68.18993232814061]
  We introduce an approach that performs 3D reconstruction and 3D panoptic segmentation in a unified framework.<n>We build on existing 3D reconstruction models and augment them with a set-based mask decoder.<n>Our method achieves state-of-the-art performance in 3D panoptic segmentation across ScanNet, ScanNet200, and ScanNet++ datasets.
  arXiv  Detail & Related papers  (2026-06-12T09:43:13Z)
• 3D-ReGen: A Unified 3D Geometry Regeneration Framework [101.26715424980996]
  We introduce 3D-ReGen, a 3D regenerator conditioned on an initial 3D shape.<n>3D-ReGen supports numerous useful tasks, including 3D enhancement, reconstruction, and editing.<n>We evaluate both the geometric consistency and fine-grained quality of 3D-ReGen, achieving state-of-the-art performance in controllable 3D generation.
  arXiv  Detail & Related papers  (2026-04-30T17:18:05Z)
• 3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image [26.04490259188974]
  We introduce 3D-Fixer, a novel in-place completion paradigm for compositional 3D scene generation.<n>Unlike prior works that require explicit pose alignment, 3D-Fixer uses fragmented geometry as a spatial anchor to preserve layout fidelity.<n>We present ARSG-110K, the largest scene-level dataset to date, comprising over 110K diverse scenes and 3M annotated images with high-fidelity 3D ground truth.
  arXiv  Detail & Related papers  (2026-04-06T04:11:09Z)
• RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations [70.83499963694238]
  RnG (Reconstruction and Generation) is a novel feed-forward Transformer that unifies reconstruction and generation.<n>It reconstructs visible geometry and generates plausible, coherent unseen geometry and appearance.<n>Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation.
  arXiv  Detail & Related papers  (2026-03-01T17:25:32Z)
• CUPID: Pose-Grounded Generative 3D Reconstruction from a Single Image [32.39661961097445]
  A new generation-based 3D reconstruction method, named Cupid, infers the camera pose, 3D shape, and texture of an object from a single 2D image.<n>Experiments demonstrate Cupid outperforms leading 3D reconstruction methods with an over 3 dB PSNR gain and an over 10% Chamfer Distance reduction.
  arXiv  Detail & Related papers  (2025-10-23T17:47:38Z)
• VIST3A: Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator [69.72818094722186]
  A text-to-video generator can be combined with a 3D reconstruction system as a "decoder"<n>We introduce VIST3A, a general framework that does just that.<n>We evaluate the proposed VIST3A approach with different video generators and 3D reconstruction models.
  arXiv  Detail & Related papers  (2025-10-15T11:55:08Z)
• Shape from Semantics: 3D Shape Generation from Multi-View Semantics [30.969299308083723]
  Existing 3D reconstruction methods utilize guidances such as 2D images, 3D point clouds, shape contours and single semantics to recover the 3D surface.<n>We propose a novel 3D modeling task called Shape from Semantics'', which aims to create 3D models whose geometry and appearance are consistent with the given text semantics when viewed from different views.
  arXiv  Detail & Related papers  (2025-02-01T07:51:59Z)
• MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model [34.245635412589806]
  MeshFormer is a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. It can be integrated with 2D diffusion models to enable fast single-image-to-3D and text-to-3D tasks.
  arXiv  Detail & Related papers  (2024-08-19T17:55:17Z)
• GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation [65.33726478659304]
  We introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms.
  arXiv  Detail & Related papers  (2024-06-21T17:49:31Z)
• NeuSDFusion: A Spatial-Aware Generative Model for 3D Shape Completion, Reconstruction, and Generation [52.772319840580074]
  3D shape generation aims to produce innovative 3D content adhering to specific conditions and constraints. Existing methods often decompose 3D shapes into a sequence of localized components, treating each element in isolation. We introduce a novel spatial-aware 3D shape generation framework that leverages 2D plane representations for enhanced 3D shape modeling.
  arXiv  Detail & Related papers  (2024-03-27T04:09:34Z)
• Fine Detailed Texture Learning for 3D Meshes with Generative Models [33.42114674602613]
  This paper presents a method to reconstruct high-quality textured 3D models from both multi-view and single-view images. In the first stage, we focus on learning accurate geometry, whereas in the second stage, we focus on learning the texture with a generative adversarial network. We demonstrate that our method achieves superior 3D textured models compared to the previous works.
  arXiv  Detail & Related papers  (2022-03-17T14:50:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.