One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing

• URL: http://arxiv.org/abs/2404.09979v1
• Date: Mon, 15 Apr 2024 17:56:05 GMT
• Title: One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing
• Authors: Yueyu Hu, Onur G. Guleryuz, Philip A. Chou, Danhang Tang, Jonathan Taylor, Rus Maxham, Yao Wang,
• Abstract summary: We propose a new approach to upgrade a 2D video to support stereo RGB-D video compression, by wrapping it with a neural pre- and post-processor pair. We train the neural pre- and post-processors on a synthetic 4D people dataset, and evaluate it on both synthetic and real-captured stereo RGB-D videos. Our approach saves about 30% bit-rate compared to a conventional video coding scheme and MV-HEVC at the same level of rendering quality from a novel view.
• Score: 13.74209129258984
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stereoscopic video conferencing is still challenging due to the need to compress stereo RGB-D video in real-time. Though hardware implementations of standard video codecs such as H.264 / AVC and HEVC are widely available, they are not designed for stereoscopic videos and suffer from reduced quality and performance. Specific multiview or 3D extensions of these codecs are complex and lack efficient implementations. In this paper, we propose a new approach to upgrade a 2D video codec to support stereo RGB-D video compression, by wrapping it with a neural pre- and post-processor pair. The neural networks are end-to-end trained with an image codec proxy, and shown to work with a more sophisticated video codec. We also propose a geometry-aware loss function to improve rendering quality. We train the neural pre- and post-processors on a synthetic 4D people dataset, and evaluate it on both synthetic and real-captured stereo RGB-D videos. Experimental results show that the neural networks generalize well to unseen data and work out-of-box with various video codecs. Our approach saves about 30% bit-rate compared to a conventional video coding scheme and MV-HEVC at the same level of rendering quality from a novel view, without the need of a task-specific hardware upgrade.

Related papers

• Fast Encoding and Decoding for Implicit Video Representation [88.43612845776265]
  We introduce NeRV-Enc, a transformer-based hyper-network for fast encoding; and NeRV-Dec, a parallel decoder for efficient video loading. NeRV-Enc achieves an impressive speed-up of $mathbf104times$ by eliminating gradient-based optimization. NeRV-Dec simplifies video decoding, outperforming conventional codecs with a loading speed $mathbf11times$ faster.
  arXiv  Detail & Related papers  (2024-09-28T18:21:52Z)
• V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians [53.614560799043545]
  V3 (Viewing Volumetric Videos) is a novel approach that enables high-quality mobile rendering through the streaming of dynamic Gaussians. Our key innovation is to view dynamic 3DGS as 2D videos, facilitating the use of hardware video codecs. As the first to stream dynamic Gaussians on mobile devices, our companion player offers users an unprecedented volumetric video experience.
  arXiv  Detail & Related papers  (2024-09-20T16:54:27Z)
• C3: High-performance and low-complexity neural compression from a single image or video [16.770509909942312]
  We introduce C3, a neural compression method with strong rate-distortion (RD) performance. The resulting decoding complexity of C3 can be an order of magnitude lower than neural baselines with similar RD performance.
  arXiv  Detail & Related papers  (2023-12-05T13:28:59Z)
• Implicit-explicit Integrated Representations for Multi-view Video Compression [40.86402535896703]
  We propose an implicit-explicit integrated representation for multi-view video compression. The proposed framework combines the strengths of both implicit neural representation and explicit 2D datasets. Our proposed framework can achieve comparable or even superior performance to the latest multi-view video compression standard MIV.
  arXiv  Detail & Related papers  (2023-11-29T04:15:57Z)
• HNeRV: A Hybrid Neural Representation for Videos [56.492309149698606]
  Implicit neural representations store videos as neural networks. We propose a Hybrid Neural Representation for Videos (HNeRV) With content-adaptive embeddings and re-designed architecture, HNeRV outperforms implicit methods in video regression tasks.
  arXiv  Detail & Related papers  (2023-04-05T17:55:04Z)
• Scalable Neural Video Representations with Learnable Positional Features [73.51591757726493]
  We show how to train neural representations with learnable positional features (NVP) that effectively amortize a video as latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07rightarrow$34.57 (measured with the PSNR metric)
  arXiv  Detail & Related papers  (2022-10-13T08:15:08Z)
• Compressed Vision for Efficient Video Understanding [83.97689018324732]
  We propose a framework enabling research on hour-long videos with the same hardware that can now process second-long videos. We replace standard video compression, e.g. JPEG, with neural compression and show that we can directly feed compressed videos as inputs to regular video networks.
  arXiv  Detail & Related papers  (2022-10-06T15:35:49Z)
• Task Oriented Video Coding: A Survey [0.5076419064097732]
  State-of-the-art video coding standards, such as H.265/HEVC and Versatile Video Coding, are still designed with the assumption the compressed video will be watched by humans. With the tremendous advance and maturation of deep neural networks in solving computer vision tasks, more and more videos are directly analyzed by deep neural networks without humans' involvement. We explore and summarize recent progress on computer vision task oriented video coding and emerging video coding standard, Video Coding for Machines.
  arXiv  Detail & Related papers  (2022-08-15T16:21:54Z)
• A Coding Framework and Benchmark towards Low-Bitrate Video Understanding [63.05385140193666]
  We propose a traditional-neural mixed coding framework that takes advantage of both traditional codecs and neural networks (NNs) The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved. We build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach.
  arXiv  Detail & Related papers  (2022-02-06T16:29:15Z)
• Efficient Video Compression via Content-Adaptive Super-Resolution [11.6624528293976]
  Video compression is a critical component of Internet video delivery. Recent work has shown that deep learning techniques can rival or outperform human algorithms. This paper presents a new approach that augments a recent deep learning-based video compression scheme.
  arXiv  Detail & Related papers  (2021-04-06T07:01:06Z)
• Learning to Compress Videos without Computing Motion [39.46212197928986]
  We propose a new deep learning video compression architecture that does not require motion estimation. Our framework exploits the regularities inherent to video motion, which we capture by using displaced frame differences as video representations. Our experiments show that our compression model, which we call the MOtionless VIdeo Codec (MOVI-Codec), learns how to efficiently compress videos without computing motion.
  arXiv  Detail & Related papers  (2020-09-29T15:49:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.