Spatial Visibility and Temporal Dynamics: Revolutionizing Field of View Prediction in Adaptive Point Cloud Video Streaming

• URL: http://arxiv.org/abs/2409.18236v2
• Date: Tue, 1 Oct 2024 21:53:44 GMT
• Title: Spatial Visibility and Temporal Dynamics: Revolutionizing Field of View Prediction in Adaptive Point Cloud Video Streaming
• Authors: Chen Li, Tongyu Zong, Yueyu Hu, Yao Wang, Yong Liu,
• Abstract summary: Field-of-View adaptive streaming significantly reduces bandwidth requirement of immersive point cloud video. Traditional approaches often focus on trajectory-based 6 degree-of-freedom (6DoF) FoV predictions. We reformulate the PCV FoV prediction problem from the cell visibility perspective.
• Score: 19.0599625095738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Field-of-View (FoV) adaptive streaming significantly reduces bandwidth requirement of immersive point cloud video (PCV) by only transmitting visible points in a viewer's FoV. The traditional approaches often focus on trajectory-based 6 degree-of-freedom (6DoF) FoV predictions. The predicted FoV is then used to calculate point visibility. Such approaches do not explicitly consider video content's impact on viewer attention, and the conversion from FoV to point visibility is often error-prone and time-consuming. We reformulate the PCV FoV prediction problem from the cell visibility perspective, allowing for precise decision-making regarding the transmission of 3D data at the cell level based on the predicted visibility distribution. We develop a novel spatial visibility and object-aware graph model that leverages the historical 3D visibility data and incorporates spatial perception, neighboring cell correlation, and occlusion information to predict the cell visibility in the future. Our model significantly improves the long-term cell visibility prediction, reducing the prediction MSE loss by up to 50% compared to the state-of-the-art models while maintaining real-time performance (more than 30fps) for point cloud videos with over 1 million points.

Related papers

• AdaOcc: Adaptive Forward View Transformation and Flow Modeling for 3D Occupancy and Flow Prediction [56.72301849123049]
  We present our solution for the Vision-Centric 3D Occupancy and Flow Prediction track in the nuScenes Open-Occ dataset challenge at CVPR 2024. Our innovative approach involves a dual-stage framework that enhances 3D occupancy and flow predictions by incorporating adaptive forward view transformation and flow modeling. Our method combines regression with classification to address scale variations in different scenes, and leverages predicted flow to warp current voxel features to future frames, guided by future frame ground truth.
  arXiv  Detail & Related papers  (2024-07-01T16:32:15Z)
• A Novel Deep Neural Network for Trajectory Prediction in Automated Vehicles Using Velocity Vector Field [12.067838086415833]
  This paper proposes a novel technique for trajectory prediction that combines a data-driven learning-based method with a velocity vector field (VVF) generated from a nature-inspired concept. The accuracy remains consistent with decreasing observation windows which alleviates the requirement of a long history of past observations for accurate trajectory prediction.
  arXiv  Detail & Related papers  (2023-09-19T22:14:52Z)
• Spherical Vision Transformer for 360-degree Video Saliency Prediction [17.948179628551376]
  We propose a vision-transformer-based model for omnidirectional videos named SalViT360. We introduce a spherical geometry-aware self-attention mechanism that is capable of effective omnidirectional video understanding. Our approach is the first to employ tangent images for omnidirectional saliency prediction prediction, and our experimental results on three ODV saliency datasets demonstrate its effectiveness compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-08-24T18:07:37Z)
• BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios [51.285561119993105]
  We present BEV-MAE, an efficient masked autoencoder pre-training framework for LiDAR-based 3D object detection in autonomous driving. Specifically, we propose a bird's eye view (BEV) guided masking strategy to guide the 3D encoder learning feature representation. We introduce a learnable point token to maintain a consistent receptive field size of the 3D encoder.
  arXiv  Detail & Related papers  (2022-12-12T08:15:03Z)
• Monocular BEV Perception of Road Scenes via Front-to-Top View Projection [57.19891435386843]
  We present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view. Our model runs at 25 FPS on a single GPU, which is efficient and applicable for real-time panorama HD map reconstruction.
  arXiv  Detail & Related papers  (2022-11-15T13:52:41Z)
• FoV-Net: Field-of-View Extrapolation Using Self-Attention and Uncertainty [95.11806655550315]
  We utilize information from a video sequence with a narrow field-of-view to infer the scene at a wider field-of-view. We propose a temporally consistent field-of-view extrapolation framework, namely FoV-Net. Experiments show that FoV-Net does not only extrapolate the temporally consistent wide field-of-view scene better than existing alternatives.
  arXiv  Detail & Related papers  (2022-04-04T06:24:03Z)
• Spherical Convolution empowered FoV Prediction in 360-degree Video Multicast with Limited FoV Feedback [16.716422953229088]
  Field of view (FoV) prediction is critical in 360-degree video multicast. This paper proposes a spherical convolution-empowered FoV prediction method. The experimental results show that the performance of the proposed method is better than other prediction methods.
  arXiv  Detail & Related papers  (2022-01-29T08:32:19Z)
• InfoFocus: 3D Object Detection for Autonomous Driving with Dynamic Information Modeling [65.47126868838836]
  We propose a novel 3D object detection framework with dynamic information modeling. Coarse predictions are generated in the first stage via a voxel-based region proposal network. Experiments are conducted on the large-scale nuScenes 3D detection benchmark.
  arXiv  Detail & Related papers  (2020-07-16T18:27:08Z)
• Deep Learning for Content-based Personalized Viewport Prediction of 360-Degree VR Videos [72.08072170033054]
  In this paper, a deep learning network is introduced to leverage position data as well as video frame content to predict future head movement. For optimizing data input into this neural network, data sample rate, reduced data, and long-period prediction length are also explored for this model.
  arXiv  Detail & Related papers  (2020-03-01T07:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.