One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing

• URL: http://arxiv.org/abs/2404.09979v1
• Date: Mon, 15 Apr 2024 17:56:05 GMT
• Title: One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing
• Authors: Yueyu Hu, Onur G. Guleryuz, Philip A. Chou, Danhang Tang, Jonathan Taylor, Rus Maxham, Yao Wang,
• Abstract summary: We propose a new approach to upgrade a 2D video to support stereo RGB-D video compression, by wrapping it with a neural pre- and post-processor pair. We train the neural pre- and post-processors on a synthetic 4D people dataset, and evaluate it on both synthetic and real-captured stereo RGB-D videos. Our approach saves about 30% bit-rate compared to a conventional video coding scheme and MV-HEVC at the same level of rendering quality from a novel view.
• Score: 13.74209129258984
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stereoscopic video conferencing is still challenging due to the need to compress stereo RGB-D video in real-time. Though hardware implementations of standard video codecs such as H.264 / AVC and HEVC are widely available, they are not designed for stereoscopic videos and suffer from reduced quality and performance. Specific multiview or 3D extensions of these codecs are complex and lack efficient implementations. In this paper, we propose a new approach to upgrade a 2D video codec to support stereo RGB-D video compression, by wrapping it with a neural pre- and post-processor pair. The neural networks are end-to-end trained with an image codec proxy, and shown to work with a more sophisticated video codec. We also propose a geometry-aware loss function to improve rendering quality. We train the neural pre- and post-processors on a synthetic 4D people dataset, and evaluate it on both synthetic and real-captured stereo RGB-D videos. Experimental results show that the neural networks generalize well to unseen data and work out-of-box with various video codecs. Our approach saves about 30% bit-rate compared to a conventional video coding scheme and MV-HEVC at the same level of rendering quality from a novel view, without the need of a task-specific hardware upgrade.

Related papers

• C3: High-performance and low-complexity neural compression from a single image or video [16.770509909942312]
  We introduce C3, a neural compression method with strong rate-distortion (RD) performance. The resulting decoding complexity of C3 can be an order of magnitude lower than neural baselines with similar RD performance.
  arXiv  Detail & Related papers  (2023-12-05T13:28:59Z)
• Implicit-explicit Integrated Representations for Multi-view Video Compression [40.86402535896703]
  We propose an implicit-explicit integrated representation for multi-view video compression. The proposed framework combines the strengths of both implicit neural representation and explicit 2D datasets. Our proposed framework can achieve comparable or even superior performance to the latest multi-view video compression standard MIV.
  arXiv  Detail & Related papers  (2023-11-29T04:15:57Z)
• HNeRV: A Hybrid Neural Representation for Videos [56.492309149698606]
  Implicit neural representations store videos as neural networks. We propose a Hybrid Neural Representation for Videos (HNeRV) With content-adaptive embeddings and re-designed architecture, HNeRV outperforms implicit methods in video regression tasks.
  arXiv  Detail & Related papers  (2023-04-05T17:55:04Z)
• Towards Scalable Neural Representation for Diverse Videos [68.73612099741956]
  Implicit neural representations (INR) have gained increasing attention in representing 3D scenes and images. Existing INR-based methods are limited to encoding a handful of short videos with redundant visual content. This paper focuses on developing neural representations for encoding long and/or a large number of videos with diverse visual content.
  arXiv  Detail & Related papers  (2023-03-24T16:32:19Z)
• MagicVideo: Efficient Video Generation With Latent Diffusion Models [76.95903791630624]
  We present an efficient text-to-video generation framework based on latent diffusion models, termed MagicVideo. Due to a novel and efficient 3D U-Net design and modeling video distributions in a low-dimensional space, MagicVideo can synthesize video clips with 256x256 spatial resolution on a single GPU card. We conduct extensive experiments and demonstrate that MagicVideo can generate high-quality video clips with either realistic or imaginary content.
  arXiv  Detail & Related papers  (2022-11-20T16:40:31Z)
• Scalable Neural Video Representations with Learnable Positional Features [73.51591757726493]
  We show how to train neural representations with learnable positional features (NVP) that effectively amortize a video as latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07rightarrow$34.57 (measured with the PSNR metric)
  arXiv  Detail & Related papers  (2022-10-13T08:15:08Z)
• Compressed Vision for Efficient Video Understanding [83.97689018324732]
  We propose a framework enabling research on hour-long videos with the same hardware that can now process second-long videos. We replace standard video compression, e.g. JPEG, with neural compression and show that we can directly feed compressed videos as inputs to regular video networks.
  arXiv  Detail & Related papers  (2022-10-06T15:35:49Z)
• Task Oriented Video Coding: A Survey [0.5076419064097732]
  State-of-the-art video coding standards, such as H.265/HEVC and Versatile Video Coding, are still designed with the assumption the compressed video will be watched by humans. With the tremendous advance and maturation of deep neural networks in solving computer vision tasks, more and more videos are directly analyzed by deep neural networks without humans' involvement. We explore and summarize recent progress on computer vision task oriented video coding and emerging video coding standard, Video Coding for Machines.
  arXiv  Detail & Related papers  (2022-08-15T16:21:54Z)
• Perceptual Coding for Compressed Video Understanding: A New Framework and Benchmark [57.23523738351178]
  We propose the first coding framework for compressed video understanding, where another learnable perceptual bitstream is introduced and simultaneously transported with the video bitstream. Our framework can enjoy the best of both two worlds, (1) highly efficient content-coding of industrial video and (2) flexible perceptual-coding of neural networks (NNs)
  arXiv  Detail & Related papers  (2022-02-06T16:29:15Z)
• Efficient Video Compression via Content-Adaptive Super-Resolution [11.6624528293976]
  Video compression is a critical component of Internet video delivery. Recent work has shown that deep learning techniques can rival or outperform human algorithms. This paper presents a new approach that augments a recent deep learning-based video compression scheme.
  arXiv  Detail & Related papers  (2021-04-06T07:01:06Z)
• Learning to Compress Videos without Computing Motion [39.46212197928986]
  We propose a new deep learning video compression architecture that does not require motion estimation. Our framework exploits the regularities inherent to video motion, which we capture by using displaced frame differences as video representations. Our experiments show that our compression model, which we call the MOtionless VIdeo Codec (MOVI-Codec), learns how to efficiently compress videos without computing motion.
  arXiv  Detail & Related papers  (2020-09-29T15:49:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.