MAC-Ego3D: Multi-Agent Gaussian Consensus for Real-Time Collaborative Ego-Motion and Photorealistic 3D Reconstruction

• URL: http://arxiv.org/abs/2412.09723v1
• Date: Thu, 12 Dec 2024 21:04:53 GMT
• Title: MAC-Ego3D: Multi-Agent Gaussian Consensus for Real-Time Collaborative Ego-Motion and Photorealistic 3D Reconstruction
• Abstract summary: We present MAC-Ego3D, a novel framework for real-time collaborative 3D reconstruction via Multi-Agent Gaussian Consensus.<n> MAC-Ego3D enables agents to independently construct, align, and iteratively refine local maps.<n>It achieves a 15x increase in inference speed, order-of-magnitude reductions in ego-motion estimation error for partial cases, and RGB PSNR gains of 4 to 10 dB.
• Score: 7.8809787820112
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time multi-agent collaboration for ego-motion estimation and high-fidelity 3D reconstruction is vital for scalable spatial intelligence. However, traditional methods produce sparse, low-detail maps, while recent dense mapping approaches struggle with high latency. To overcome these challenges, we present MAC-Ego3D, a novel framework for real-time collaborative photorealistic 3D reconstruction via Multi-Agent Gaussian Consensus. MAC-Ego3D enables agents to independently construct, align, and iteratively refine local maps using a unified Gaussian splat representation. Through Intra-Agent Gaussian Consensus, it enforces spatial coherence among neighboring Gaussian splats within an agent. For global alignment, parallelized Inter-Agent Gaussian Consensus, which asynchronously aligns and optimizes local maps by regularizing multi-agent Gaussian splats, seamlessly integrates them into a high-fidelity 3D model. Leveraging Gaussian primitives, MAC-Ego3D supports efficient RGB-D rendering, enabling rapid inter-agent Gaussian association and alignment. MAC-Ego3D bridges local precision and global coherence, delivering higher efficiency, largely reducing localization error, and improving mapping fidelity. It establishes a new SOTA on synthetic and real-world benchmarks, achieving a 15x increase in inference speed, order-of-magnitude reductions in ego-motion estimation error for partial cases, and RGB PSNR gains of 4 to 10 dB. Our code will be made publicly available at https://github.com/Xiaohao-Xu/MAC-Ego3D .

Related papers

• MAC-Splat: Multi-Attribute Consistency for High-Fidelity Sparse-View Reconstruction [22.10774833540097]
  We propose a training framework built around direct 3D consistency supervision.<n> MAC-Splat builds on the MASt3R geometric backbone and a frozen DINOv3 encoder to obtain semantically informed 2D correspondences.<n>Experiments on ScanNet++ show that MAC-Splat outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-07-12T14:50:08Z)
• Splaxel: Efficient Distributed Training of 3D Gaussian Splatting for Large-scale Scene Reconstruction via Pixel-level Communication [23.72228306115471]
  Splaxel is a communication-efficient 3DGS training framework based on pixel-level local rendering and global composition.<n>Spaxel achieves up to 7.6$times$ speedup over the state-of-the-art distributed 3DGS framework.
  arXiv  Detail & Related papers  (2026-06-17T01:27:41Z)
• Local-GS: Accelerating 3D Gaussian Splatting via Tile-Local Warp Coherence [31.001161060131622]
  3D Gaussian Splatting (3DGS) has significantly advanced real-time novel view computation by representing scenes as dense collections of anisotropic 3D Gaussian primitives.<n>We present Local-GS, a warp-coherent rendering paradigm that organizes Gaussian primitives with respect to SIMT (Single Instruction, Multiple Threads) execution boundaries rather than scene geometry.<n>As a plug-and-play optimization, it provides additional performance gains to all tested baselines, culminating in a $7.76times$ speedup on Deep Blending scenes.
  arXiv  Detail & Related papers  (2026-06-15T11:08:10Z)
• Compact Keyframe-Optimized Multi-Agent Gaussian Splatting SLAM [4.458834881246724]
  Multi-agent 3D mapping is essential for robotic teams operating in unknown environments, but dense representations hinder real-time exchange.<n>We present an improved multi-agent RGB-D Gaussian Splatting SLAM framework that reduces communication load while preserving map fidelity.
  arXiv  Detail & Related papers  (2026-04-01T12:11:20Z)
• Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding [86.55824709875598]
  We propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches.<n>Unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor to capture fine-grained 3D shape details.<n>We employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations.
  arXiv  Detail & Related papers  (2026-01-05T18:33:50Z)
• MAGiC-SLAM: Multi-Agent Gaussian Globally Consistent SLAM [23.318966306555915]
  Simultaneous localization and mapping (SLAM) systems are widely used in computer vision, with applications in augmented reality, robotics, and autonomous driving. Recent work has addressed this problem using a distributed neural scene representation. We propose a rigidly deformable 3D Gaussian-based scene representation that dramatically speeds up the system. We evaluate MAGiC-SLAM on synthetic and real-world datasets and find it more accurate and faster than the state of the art.
  arXiv  Detail & Related papers  (2024-11-25T08:34:01Z)
• MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields [100.90743697473232]
  Radiance fields represented by 3D Gaussians excel at synthesizing novel views, offering both high training efficiency and fast rendering.<n>Existing methods often incorporate depth priors from dense estimation networks but overlook the inherent multi-view consistency in input images.<n>We propose a view synthesis framework based on 3D Gaussian Splatting, enabling scene reconstruction from sparse views.
  arXiv  Detail & Related papers  (2024-10-15T08:39:05Z)
• Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency [6.119688280076556]
  3D Gaussians (3DGS) have proven a versatile rendering primitive, both for inverse rendering and real-time exploration of scenes. Recent work started mitigating artifacts that break multi-view coherence, including popping artifacts due to inconsistent transparency sorting and perspective-correct outlines of (2D) splats. In our work, we aim at achieving maximum coherence, by rendering fully perspective-correct 3D Gaussians while using a high-quality approximation of accurate blending, hybrid transparency, on a per-pixel level, in order to retain real-time frame rates.
  arXiv  Detail & Related papers  (2024-10-10T17:14:16Z)
• GSFusion: Online RGB-D Mapping Where Gaussian Splatting Meets TSDF Fusion [12.964675001994124]
  Traditional fusion algorithms preserve the spatial structure of 3D scenes. They often lack realism in terms of visualization. GSFusion significantly enhances computational efficiency without sacrificing rendering quality.
  arXiv  Detail & Related papers  (2024-08-22T18:32:50Z)
• PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting [59.277480452459315]
  We propose a principled sensitivity pruning score that preserves visual fidelity and foreground details at significantly higher compression ratios. We also propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model without changing its training pipeline.
  arXiv  Detail & Related papers  (2024-06-14T17:53:55Z)
• GSGAN: Adversarial Learning for Hierarchical Generation of 3D Gaussian Splats [20.833116566243408]
  In this paper, we exploit Gaussian as a 3D representation for 3D GANs by leveraging its efficient and explicit characteristics. We introduce a generator architecture with a hierarchical multi-scale Gaussian representation that effectively regularizes the position and scale of generated Gaussians. Experimental results demonstrate that ours achieves a significantly faster rendering speed (x100) compared to state-of-the-art 3D consistent GANs.
  arXiv  Detail & Related papers  (2024-06-05T05:52:20Z)
• Splat-SLAM: Globally Optimized RGB-only SLAM with 3D Gaussians [87.48403838439391]
  3D Splatting has emerged as a powerful representation of geometry and appearance for RGB-only dense Simultaneous SLAM. We propose the first RGB-only SLAM system with a dense 3D Gaussian map representation. Our experiments on the Replica, TUM-RGBD, and ScanNet datasets indicate the effectiveness of globally optimized 3D Gaussians.
  arXiv  Detail & Related papers  (2024-05-26T12:26:54Z)
• RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting [51.51310922527121]
  We present a real-time 3D reconstruction system with an RGBD camera for large-scale environments using Gaussian splatting. We force each Gaussian to be either opaque or nearly transparent, with the opaque ones fitting the surface and dominant colors, and transparent ones fitting residual colors. We show real-time reconstructions of a variety of large scenes and show superior performance in the realism of novel view synthesis and camera tracking accuracy.
  arXiv  Detail & Related papers  (2024-04-30T16:54:59Z)
• CompGS: Efficient 3D Scene Representation via Compressed Gaussian Splatting [68.94594215660473]
  We propose an efficient 3D scene representation, named Compressed Gaussian Splatting (CompGS) We exploit a small set of anchor primitives for prediction, allowing the majority of primitives to be encapsulated into highly compact residual forms. Experimental results show that the proposed CompGS significantly outperforms existing methods, achieving superior compactness in 3D scene representation without compromising model accuracy and rendering quality.
  arXiv  Detail & Related papers  (2024-04-15T04:50:39Z)
• latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction [48.86083272054711]
  latentSplat is a method to predict semantic Gaussians in a 3D latent space that can be splatted and decoded by a light-weight generative 2D architecture. We show that latentSplat outperforms previous works in reconstruction quality and generalization, while being fast and scalable to high-resolution data.
  arXiv  Detail & Related papers  (2024-03-24T20:48:36Z)
• SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM [48.190398577764284]
  SplaTAM is an approach to enable high-fidelity reconstruction from a single unposed RGB-D camera. It employs a simple online tracking and mapping system tailored to the underlying Gaussian representation. Experiments show that SplaTAM achieves up to 2x superior performance in camera pose estimation, map construction, and novel-view synthesis over existing methods.
  arXiv  Detail & Related papers  (2023-12-04T18:53:24Z)
• GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting [51.96353586773191]
  We introduce textbfGS-SLAM that first utilizes 3D Gaussian representation in the Simultaneous Localization and Mapping system. Our method utilizes a real-time differentiable splatting rendering pipeline that offers significant speedup to map optimization and RGB-D rendering. Our method achieves competitive performance compared with existing state-of-the-art real-time methods on the Replica, TUM-RGBD datasets.
  arXiv  Detail & Related papers  (2023-11-20T12:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.