Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras

• URL: http://arxiv.org/abs/2405.14866v1
• Date: Thu, 23 May 2024 17:59:45 GMT
• Title: Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras
• Abstract summary: We present a low-budget and high-authenticity bidirectional telepresence system, Tele-Aloha, targeting peer-to-peer communication scenarios. Tele-Aloha utilizes only four sparse RGB cameras, one consumer-grade GPU, and one autostereoscopic screen to achieve high-resolution (2048x2048), real-time (30 fps), low-latency (less than 150ms) and robust distant communication.
• Score: 34.520964056321105
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we present a low-budget and high-authenticity bidirectional telepresence system, Tele-Aloha, targeting peer-to-peer communication scenarios. Compared to previous systems, Tele-Aloha utilizes only four sparse RGB cameras, one consumer-grade GPU, and one autostereoscopic screen to achieve high-resolution (2048x2048), real-time (30 fps), low-latency (less than 150ms) and robust distant communication. As the core of Tele-Aloha, we propose an efficient novel view synthesis algorithm for upper-body. Firstly, we design a cascaded disparity estimator for obtaining a robust geometry cue. Additionally a neural rasterizer via Gaussian Splatting is introduced to project latent features onto target view and to decode them into a reduced resolution. Further, given the high-quality captured data, we leverage weighted blending mechanism to refine the decoded image into the final resolution of 2K. Exploiting world-leading autostereoscopic display and low-latency iris tracking, users are able to experience a strong three-dimensional sense even without any wearable head-mounted display device. Altogether, our telepresence system demonstrates the sense of co-presence in real-life experiments, inspiring the next generation of communication.

Related papers

• Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras [43.56045236607455]
  We present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization.<n>Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance.<n>Experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU.
  arXiv  Detail & Related papers  (2026-09-14T04:50:52Z)
• High-Resolution Single-Shot Polarimetric Imaging Made Easy [76.04914313843774]
  EasyPolar is a multi-view polarimetric imaging framework.<n>Three synchronized RGB cameras capture one unpolarized view and two polarized views with distinct orientations.<n>A confidence-guided polarization reconstruction network is proposed to address the potential misalignment in multi-view fusion.
  arXiv  Detail & Related papers  (2026-04-07T08:20:58Z)
• A Multi-View 3D Telepresence System for XR Robot Teleoperation [50.624253593914084]
  effective teleoperation requires intuitive 3D visualization with reliable depth cues.<n>We introduce a multi-view VR telepresence system that fuses geometry from three cameras to produce GPU-accelerated point-cloud rendering on standalone VR hardware.<n>Our pipeline supports real-time rendering of approximately 75k points on the Meta Quest 3.
  arXiv  Detail & Related papers  (2026-04-04T13:22:12Z)
• Mon3tr: Monocular 3D Telepresence with Pre-built Gaussian Avatars as Amortization [16.68162021163563]
  Mon3tr is a novel Monocular 3D telepresence framework that integrates 3D Gaussian splatting (3DGS) based parametric human modeling.<n>A single monocular RGB camera is used to capture body motions and facial expressions in real time to drive the 3DGS-based parametric human model.<n>Our method achieves a PSNR of > 28 dB for novel poses, an end-to-end latency of 80 ms, and > 1000x bandwidth reduction compared to point-cloud streaming.
  arXiv  Detail & Related papers  (2026-01-12T13:17:41Z)
• DiFuse-Net: RGB and Dual-Pixel Depth Estimation using Window Bi-directional Parallax Attention and Cross-modal Transfer Learning [1.456352735394398]
  DiFuse-Net is a novel decoupled network design for disentangled RGB and DP based depth estimation.<n>WBiPAM captures subtle DP disparity cues unique to smartphone cameras with small aperture.<n>A separate encoder extracts contextual information from the RGB image, and these features are fused to enhance depth prediction.
  arXiv  Detail & Related papers  (2025-06-17T16:49:27Z)
• PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting [38.47270795377126]
  We present PanSplat, a generalizable, feed-forward approach that efficiently supports resolution up to 4K (2048 $times$ 4096) Our approach features a tailored spherical 3D Gaussian pyramid with a Fibonacci lattice arrangement, enhancing image quality while reducing information redundancy. Experiments demonstrate that PanSplat achieves superior efficiency and image quality across both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-12-16T18:59:45Z)
• VR-NeRF: High-Fidelity Virtualized Walkable Spaces [55.51127858816994]
  We present an end-to-end system for the high-fidelity capture, model reconstruction, and real-time rendering of walkable spaces in virtual reality using neural radiance fields.
  arXiv  Detail & Related papers  (2023-11-05T02:03:14Z)
• Unsupervised OmniMVS: Efficient Omnidirectional Depth Inference via Establishing Pseudo-Stereo Supervision [40.58193195996798]
  We propose the first unsupervised omnidirectional MVS framework based on multiple fisheye images. The two 360deg images formulate a stereo pair with a special pose, and the photometric consistency is leveraged to establish the unsupervised constraint. Experiments exhibit that the performance of our unsupervised solution is competitive to that of the state-of-the-art supervised methods.
  arXiv  Detail & Related papers  (2023-02-20T11:35:55Z)
• A Flexible Framework for Virtual Omnidirectional Vision to Improve Operator Situation Awareness [2.817412580574242]
  We present a flexible framework for virtual projections to increase situation awareness based on a novel method to fuse multiple cameras mounted anywhere on the robot. We propose a complementary approach to improve scene understanding by fusing camera images and geometric 3D Lidar data to obtain a colorized point cloud.
  arXiv  Detail & Related papers  (2023-02-01T10:40:05Z)
• Dynamic Neural Portraits [58.480811535222834]
  We present Dynamic Neural Portraits, a novel approach to the problem of full-head reenactment. Our method generates photo-realistic video portraits by explicitly controlling head pose, facial expressions and eye gaze. Our experiments demonstrate that the proposed method is 270 times faster than recent NeRF-based reenactment methods.
  arXiv  Detail & Related papers  (2022-11-25T10:06:14Z)
• StereoVoxelNet: Real-Time Obstacle Detection Based on Occupancy Voxels from a Stereo Camera Using Deep Neural Networks [32.7826524859756]
  Obstacle detection is a safety-critical problem in robot navigation, where stereo matching is a popular vision-based approach. This paper proposes a computationally efficient method that leverages a deep neural network to detect occupancy from stereo images directly. Our approach detects obstacles accurately in the range of 32 meters and achieves better IoU (Intersection over Union) and CD (Chamfer Distance) scores with only 2% of the computation cost of the state-of-the-art stereo model.
  arXiv  Detail & Related papers  (2022-09-18T03:32:38Z)
• Neural Disparity Refinement for Arbitrary Resolution Stereo [67.55946402652778]
  We introduce a novel architecture for neural disparity refinement aimed at facilitating deployment of 3D computer vision on cheap and widespread consumer devices. Our approach relies on a continuous formulation that enables to estimate a refined disparity map at any arbitrary output resolution.
  arXiv  Detail & Related papers  (2021-10-28T18:00:00Z)
• SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera [97.0162841635425]
  We present a solution to egocentric 3D body pose estimation from monocular images captured from downward looking fish-eye cameras installed on the rim of a head mounted VR device. This unusual viewpoint leads to images with unique visual appearance, with severe self-occlusions and perspective distortions. We propose an encoder-decoder architecture with a novel multi-branch decoder designed to account for the varying uncertainty in 2D predictions.
  arXiv  Detail & Related papers  (2020-11-02T16:18:06Z)
• Active Perception with A Monocular Camera for Multiscopic Vision [50.370074098619185]
  We design a multiscopic vision system that utilizes a low-cost monocular RGB camera to acquire accurate depth estimation for robotic applications. Unlike multi-view stereo with images captured at unconstrained camera poses, the proposed system actively controls a robot arm with a mounted camera to capture a sequence of images in horizontally or vertically aligned positions with the same parallax.
  arXiv  Detail & Related papers  (2020-01-22T08:46:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.