Mon3tr: Monocular 3D Telepresence with Pre-built Gaussian Avatars as Amortization

• URL: http://arxiv.org/abs/2601.07518v1
• Date: Mon, 12 Jan 2026 13:17:41 GMT
• Title: Mon3tr: Monocular 3D Telepresence with Pre-built Gaussian Avatars as Amortization
• Abstract summary: Mon3tr is a novel Monocular 3D telepresence framework that integrates 3D Gaussian splatting (3DGS) based parametric human modeling.<n>A single monocular RGB camera is used to capture body motions and facial expressions in real time to drive the 3DGS-based parametric human model.<n>Our method achieves a PSNR of > 28 dB for novel poses, an end-to-end latency of 80 ms, and > 1000x bandwidth reduction compared to point-cloud streaming.
• Score: 16.68162021163563
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Immersive telepresence aims to transform human interaction in AR/VR applications by enabling lifelike full-body holographic representations for enhanced remote collaboration. However, existing systems rely on hardware-intensive multi-camera setups and demand high bandwidth for volumetric streaming, limiting their real-time performance on mobile devices. To overcome these challenges, we propose Mon3tr, a novel Monocular 3D telepresence framework that integrates 3D Gaussian splatting (3DGS) based parametric human modeling into telepresence for the first time. Mon3tr adopts an amortized computation strategy, dividing the process into a one-time offline multi-view reconstruction phase to build a user-specific avatar and a monocular online inference phase during live telepresence sessions. A single monocular RGB camera is used to capture body motions and facial expressions in real time to drive the 3DGS-based parametric human model, significantly reducing system complexity and cost. The extracted motion and appearance features are transmitted at < 0.2 Mbps over WebRTC's data channel, allowing robust adaptation to network fluctuations. On the receiver side, e.g., Meta Quest 3, we develop a lightweight 3DGS attribute deformation network to dynamically generate corrective 3DGS attribute adjustments on the pre-built avatar, synthesizing photorealistic motion and appearance at ~ 60 FPS. Extensive experiments demonstrate the state-of-the-art performance of our method, achieving a PSNR of > 28 dB for novel poses, an end-to-end latency of ~ 80 ms, and > 1000x bandwidth reduction compared to point-cloud streaming, while supporting real-time operation from monocular inputs across diverse scenarios. Our demos can be found at https://mon3tr3d.github.io.

Related papers

• Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras [43.56045236607455]
  We present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization.<n>Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance.<n>Experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU.
  arXiv  Detail & Related papers  (2026-09-14T04:50:52Z)
• Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors [54.51965288351254]
  We present an online 3D vision-language model that enables real-time spatial understanding from streaming video.<n>Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction to learn when to respond.<n>To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline.
  arXiv  Detail & Related papers  (2026-06-05T04:16:24Z)
• RobotPan: A 360$^\circ$ Surround-View Robotic Vision System for Embodied Perception [47.76543396190029]
  We introduce a surround-view robotic vision system that combines six cameras with LiDAR to provide full 360$circ$ visual coverage.<n>We also present textscRobotPan, a feed-forward framework that predicts emphmetric-scaled and emphcompact 3D Gaussians from calibrated sparse-view inputs.<n>Experiments show that textscRobotPan achieves competitive quality against prior feed-forward reconstruction and view-synthesis methods.
  arXiv  Detail & Related papers  (2026-04-15T04:58:23Z)
• A Multi-View 3D Telepresence System for XR Robot Teleoperation [50.624253593914084]
  effective teleoperation requires intuitive 3D visualization with reliable depth cues.<n>We introduce a multi-view VR telepresence system that fuses geometry from three cameras to produce GPU-accelerated point-cloud rendering on standalone VR hardware.<n>Our pipeline supports real-time rendering of approximately 75k points on the Meta Quest 3.
  arXiv  Detail & Related papers  (2026-04-04T13:22:12Z)
• GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System [52.652341168561016]
  We propose GSStream, a novel volumetric scene streaming system to support 3DGS data format.<n> GSStream integrates a collaborative viewport prediction module to better predict users' future behaviors.<n>Our proposed GSStream system outperforms existing representative volumetric streaming systems in visual quality and network usage.
  arXiv  Detail & Related papers  (2026-03-10T14:25:00Z)
• Stereo-Inertial Poser: Towards Metric-Accurate Shape-Aware Motion Capture Using Sparse IMUs and a Single Stereo Camera [54.967647497048205]
  We present Stereo-Inertial Poser, a real-time motion capture system that estimates metric-accurate and shape-aware 3D human motion.<n>We replace the monocular RGB with stereo vision, enabling direct 3D keypoint extraction and body shape parameter estimation.<n>Our method produces drift-free global translation under a long recording time and reduces foot-skating effects.
  arXiv  Detail & Related papers  (2026-03-02T17:46:38Z)
• Audio Driven Real-Time Facial Animation for Social Telepresence [65.66220599734338]
  We present an audio-driven real-time system for animating photorealistic 3D facial avatars with minimal latency.<n>Central to our approach is an encoder model that transforms audio signals into latent facial expression sequences in real time.<n>We capture the rich spectrum of facial expressions necessary for natural communication while achieving real-time performance.
  arXiv  Detail & Related papers  (2025-10-01T17:57:05Z)
• M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation [65.48046909056468]
  We reformulate talking head generation into a unified framework comprising video preprocessing, motion representation, and rendering reconstruction.<n>M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness.
  arXiv  Detail & Related papers  (2025-07-11T04:48:12Z)
• Progressive Inertial Poser: Progressive Real-Time Kinematic Chain Estimation for 3D Full-Body Pose from Three IMU Sensors [25.67875816218477]
  Full-body pose estimation from sparse tracking signals is not limited by environmental conditions or recording range.<n>Previous works either face the challenge of wearing additional sensors on the pelvis and lower-body or rely on external visual sensors to obtain global positions of key joints.<n>To improve the practicality of the technology for virtual reality applications, we estimate full-body poses using only inertial data obtained from three Inertial Measurement Unit (IMU) sensors worn on the head and wrists.
  arXiv  Detail & Related papers  (2025-05-08T15:28:09Z)
• EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis [61.1662426227688]
  Existing NeRF and 3DGS-based methods show promising results in achieving photorealistic renderings but require slow, per-scene optimization.<n>We introduce EVolSplat, an efficient 3D Gaussian Splatting model for urban scenes that works in a feed-forward manner.
  arXiv  Detail & Related papers  (2025-03-26T02:47:27Z)
• TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting [4.011241510647248]
  We present TaoAvatar, a high-fidelity, lightweight, 3DGS-based full-body talking avatar driven by various signals.<n>We show that TaoAvatar achieves state-of-the-art rendering quality while running in real-time across various devices, maintaining 90 FPS on high-definition stereo devices such as the Apple Vision Pro.
  arXiv  Detail & Related papers  (2025-03-21T10:40:37Z)
• SwinGS: Sliding Window Gaussian Splatting for Volumetric Video Streaming with Arbitrary Length [2.4844080708094745]
  This paper introduces SwinGS, a novel framework for training, delivering, and rendering volumetric video in a real-time streaming fashion.<n>We implement a prototype of SwinGS and demonstrate its streamability across various datasets and scenes.<n>We also develop an interactive WebGL viewer enabling real-time volumetric video playback on most devices with modern browsers.
  arXiv  Detail & Related papers  (2024-09-12T05:33:15Z)
• EmbodiedSAM: Online Segment Any 3D Thing in Real Time [61.2321497708998]
  Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration.<n>An online, real-time, fine-grained and highly-generalized 3D perception model is desperately needed.
  arXiv  Detail & Related papers  (2024-08-21T17:57:06Z)
• WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collections [8.261637198675151]
  Novel View Synthesis (NVS) from unconstrained photo collections is challenging in computer graphics. We propose an efficient point-based differentiable rendering framework for scene reconstruction from photo collections. Our approach outperforms existing approaches on the rendering quality of novel view and appearance synthesis with high converge and rendering speed.
  arXiv  Detail & Related papers  (2024-06-04T15:17:37Z)
• InstantSplat: Sparse-view Gaussian Splatting in Seconds [91.77050739918037]
  We introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed.<n>InstantSplat employs a self-supervised framework that optimize 3D scene representation and camera poses.<n>It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS.
  arXiv  Detail & Related papers  (2024-03-29T17:29:58Z)
• Multi-view data capture for dynamic object reconstruction using handheld augmented reality mobiles [0.0]
  We propose a system to capture nearly-synchronous frame streams from multiple and moving handheld mobiles. Each mobile executes Simultaneous Localisation and Mapping on-board to estimate its pose, and uses a wireless communication channel to send or receive synchronisation triggers. We show the effectiveness of our system by employing it for 3D skeleton and volumetric reconstructions.
  arXiv  Detail & Related papers  (2021-03-14T10:26:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.