DeformStream: Deformation-based Adaptive Volumetric Video Streaming

• URL: http://arxiv.org/abs/2409.16615v1
• Date: Wed, 25 Sep 2024 04:43:59 GMT
• Title: DeformStream: Deformation-based Adaptive Volumetric Video Streaming
• Authors: Boyan Li, Yongting Chen, Dayou Zhang, Fangxin Wang,
• Abstract summary: Volumetric video streaming offers immersive 3D experiences but faces significant challenges due to high bandwidth requirements and latency issues. We introduce Deformation-based Adaptive Volumetric Video Streaming, a novel framework that enhances volumetric video streaming performance by leveraging the inherent deformability of mesh-based representations.
• Score: 4.366356163044466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Volumetric video streaming offers immersive 3D experiences but faces significant challenges due to high bandwidth requirements and latency issues in transmitting detailed content in real time. Traditional methods like point cloud streaming compromise visual quality when zoomed in, and neural rendering techniques are too computationally intensive for real-time use. Though mesh-based streaming stands out by preserving surface detail and connectivity, offering a more refined representation for 3D content, traditional mesh streaming methods typically transmit data on a per-frame basis, failing to take full advantage of temporal redundancies across frames. This results in inefficient bandwidth usage and poor adaptability to fluctuating network conditions. We introduce Deformation-based Adaptive Volumetric Video Streaming, a novel framework that enhances volumetric video streaming performance by leveraging the inherent deformability of mesh-based representations. DeformStream uses embedded deformation to reconstruct subsequent frames from inter-frame motion, significantly reducing bandwidth usage while ensuring visual coherence between frames. To address frame reconstruction overhead and network adaptability, we formulate a new QoE model that accounts for client-side deformation latency and design a dynamic programming algorithm to optimize the trade-off between visual quality and bandwidth consumption under varying network conditions. Our evaluation demonstrates that Deformation-based Adaptive Volumetric Video Streaming outperforms existing mesh-based streaming systems in both bandwidth efficiency and visual quality, offering a robust solution for real-time volumetric video applications.

Related papers

• Semantic-Aware Adaptive Video Streaming Using Latent Diffusion Models for Wireless Networks [12.180483357502293]
  This paper proposes a novel framework for real-time adaptive-bitrate video streaming by integrating latent diffusion models (LDMs) within the FF techniques. It addresses the challenges of high bandwidth usage, storage inefficiencies, and quality of experience (QoE) associated with traditional constant streaming (CBS) and adaptive streaming (ABS) This work opens new scalable solutions for real-time video streaming in 5G and future post-5G networks.
  arXiv  Detail & Related papers  (2025-02-08T21:14:28Z)
• StreamChat: Chatting with Streaming Video [85.02875830683637]
  StreamChat is a novel approach that enhances the interaction capabilities of Large Multimodal Models with streaming video content. We introduce a flexible and efficient crossattention-based architecture to process dynamic streaming inputs. We construct a new dense instruction dataset to facilitate the training of streaming interaction models.
  arXiv  Detail & Related papers  (2024-12-11T18:59:54Z)
• CD-NGP: A Fast Scalable Continual Representation for Dynamic Scenes [31.783117836434403]
  Current methods for novel view synthesis (NVS) in dynamic scenes encounter significant challenges in managing memory consumption, model complexity, training efficiency, and rendering fidelity. We propose continual dynamic neural graphics primitives (CD-NGP) to address these issues. Our approach leverages a continual learning framework to reduce memory overhead, and it also integrates features from distinct temporal and spatial hash encodings for high rendering quality.
  arXiv  Detail & Related papers  (2024-09-08T17:35:48Z)
• FrameCorr: Adaptive, Autoencoder-based Neural Compression for Video Reconstruction in Resource and Timing Constrained Network Settings [0.18906710320196732]
  Existing video compression methods face difficulties in recovering compressed data when incomplete data is provided. We introduce FrameCorr, a deep-learning based solution that utilizes previously received data to predict the missing segments of a frame.
  arXiv  Detail & Related papers  (2024-09-04T05:19:57Z)
• Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams [78.72965584414368]
  We present Flash-VStream, a video-language model that simulates the memory mechanism of human. Compared to existing models, Flash-VStream achieves significant reductions in latency inference and VRAM consumption. We propose VStream-QA, a novel question answering benchmark specifically designed for online video streaming understanding.
  arXiv  Detail & Related papers  (2024-06-12T11:07:55Z)
• Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements [83.5820690348833]
  We present a framework for low-level vision tasks that does not require any external training data corpus. Our approach learns neural modules by optimizing over a corrupted sequence, leveraging the weights of the coherence-temporal test and statistics internal statistics.
  arXiv  Detail & Related papers  (2023-12-13T01:57:11Z)
• Streaming Anchor Loss: Augmenting Supervision with Temporal Significance [5.7654216719335105]
  Streaming neural network models for fast frame-wise responses to various speech and sensory signals are widely adopted on resource-constrained platforms. We propose a new loss, Streaming Anchor Loss (SAL), to better utilize the given learning capacity by encouraging the model to learn more from essential frames.
  arXiv  Detail & Related papers  (2023-10-09T17:28:35Z)
• Neural Progressive Meshes [54.52990060976026]
  We propose a method to transmit 3D meshes with a shared learned generative space. We learn this space using a subdivision-based encoder-decoder architecture trained in advance on a large collection of surfaces. We evaluate our method on a diverse set of complex 3D shapes and demonstrate that it outperforms baselines in terms of compression ratio and reconstruction quality.
  arXiv  Detail & Related papers  (2023-08-10T17:58:02Z)
• Online Streaming Video Super-Resolution with Convolutional Look-Up Table [26.628925884353674]
  This paper focuses on the rarely exploited problem setting of online streaming video super resolution. New benchmark dataset named LDV-WebRTC is constructed based on a real-world online streaming system. We propose a mixture-of-expert-LUT module, where a set of LUT specialized in different degradations are built and adaptively combined to handle different degradations.
  arXiv  Detail & Related papers  (2023-03-01T08:54:56Z)
• Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation [49.17930380106643]
  We propose a novel framework to speed up any architecture with skip-connections for real-time vision tasks. Specifically, at the arrival of each frame, we transform the features from the previous frame to reuse them at specific spatial bins. We then perform partial computation of the backbone network on the regions of the current frame that captures temporal differences between the current and previous frame.
  arXiv  Detail & Related papers  (2022-06-20T07:20:02Z)
• Capturing Video Frame Rate Variations via Entropic Differencing [63.749184706461826]
  We propose a novel statistical entropic differencing method based on a Generalized Gaussian Distribution model. Our proposed model correlates very well with subjective scores in the recently proposed LIVE-YT-HFR database.
  arXiv  Detail & Related papers  (2020-06-19T22:16:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.