GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting

• URL: http://arxiv.org/abs/2501.12060v2
• Date: Wed, 22 Jan 2025 17:24:38 GMT
• Title: GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting
• Authors: Longan Wang, Yuang Shi, Wei Tsang Ooi,
• Abstract summary: We propose GSVC, an approach to learning a set of 2D Gaussian splats that can effectively represent and compress video frames. Experiment results show that GSVC achieves good rate-distortion trade-offs, comparable to state-of-the-art video codecs.
• Score: 3.479384894190067
• License:
• Abstract: 3D Gaussian splats have emerged as a revolutionary, effective, learned representation for static 3D scenes. In this work, we explore using 2D Gaussian splats as a new primitive for representing videos. We propose GSVC, an approach to learning a set of 2D Gaussian splats that can effectively represent and compress video frames. GSVC incorporates the following techniques: (i) To exploit temporal redundancy among adjacent frames, which can speed up training and improve the compression efficiency, we predict the Gaussian splats of a frame based on its previous frame; (ii) To control the trade-offs between file size and quality, we remove Gaussian splats with low contribution to the video quality; (iii) To capture dynamics in videos, we randomly add Gaussian splats to fit content with large motion or newly-appeared objects; (iv) To handle significant changes in the scene, we detect key frames based on loss differences during the learning process. Experiment results show that GSVC achieves good rate-distortion trade-offs, comparable to state-of-the-art video codecs such as AV1 and VVC, and a rendering speed of 1500 fps for a 1920x1080 video.

Related papers

• Large Motion Video Autoencoding with Cross-modal Video VAE [52.13379965800485]
  Video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation. Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance. We present a novel and powerful video autoencoder capable of high-fidelity video encoding.
  arXiv  Detail & Related papers  (2024-12-23T18:58:24Z)
• Representing Long Volumetric Video with Temporal Gaussian Hierarchy [80.51373034419379]
  This paper aims to address the challenge of reconstructing long volumetric videos from multi-view RGB videos. We propose a novel 4D representation, named Temporal Gaussian Hierarchy, to compactly model long volumetric videos. This work is the first approach capable of efficiently handling minutes of volumetric video data while maintaining state-of-the-art rendering quality.
  arXiv  Detail & Related papers  (2024-12-12T18:59:34Z)
• LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors [107.83398512719981]
  Single-image 3D reconstruction remains a fundamental challenge in computer vision. Recent advances in Latent Video Diffusion Models offer promising 3D priors learned from large-scale video data. We propose LiftImage3D, a framework that effectively releases LVDMs' generative priors while ensuring 3D consistency.
  arXiv  Detail & Related papers  (2024-12-12T18:58:42Z)
• HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting [7.507657419706855]
  This paper proposes an efficient framework, dubbed HiCoM, with three key components. First, we construct a compact and robust initial 3DGS representation using a perturbation smoothing strategy. Next, we introduce a Hierarchical Coherent Motion mechanism that leverages the inherent non-uniform distribution and local consistency of 3D Gaussians. Experiments conducted on two widely used datasets show that our framework improves learning efficiency of the state-of-the-art methods by about $20%$.
  arXiv  Detail & Related papers  (2024-11-12T04:40:27Z)
• V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians [53.614560799043545]
  V3 (Viewing Volumetric Videos) is a novel approach that enables high-quality mobile rendering through the streaming of dynamic Gaussians. Our key innovation is to view dynamic 3DGS as 2D videos, facilitating the use of hardware video codecs. As the first to stream dynamic Gaussians on mobile devices, our companion player offers users an unprecedented volumetric video experience.
  arXiv  Detail & Related papers  (2024-09-20T16:54:27Z)
• Compact 3D Gaussian Splatting for Static and Dynamic Radiance Fields [13.729716867839509]
  We propose a learnable mask strategy that significantly reduces the number of Gaussians while preserving high performance. In addition, we propose a compact but effective representation of view-dependent color by employing a grid-based neural field. Our work provides a comprehensive framework for 3D scene representation, achieving high performance, fast training, compactness, and real-time rendering.
  arXiv  Detail & Related papers  (2024-08-07T14:56:34Z)
• Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splatting [94.84688557937123]
  Video-3DGS is a 3D Gaussian Splatting (3DGS)-based video refiner designed to enhance temporal consistency in zero-shot video editors. Our approach utilizes a two-stage 3D Gaussian optimizing process tailored for editing dynamic monocular videos. It enhances video editing by ensuring temporal consistency across 58 dynamic monocular videos.
  arXiv  Detail & Related papers  (2024-06-04T17:57:37Z)
• ContextGS: Compact 3D Gaussian Splatting with Anchor Level Context Model [77.71796503321632]
  We introduce a context model in the anchor level for 3DGS representation, yielding an impressive size reduction of over 100 times compared to vanilla 3DGS. Our work pioneers the context model in the anchor level for 3DGS representation, yielding an impressive size reduction of over 100 times compared to vanilla 3DGS and 15 times compared to the most recent state-of-the-art work Scaffold-GS.
  arXiv  Detail & Related papers  (2024-05-31T09:23:39Z)
• Predictive Coding For Animation-Based Video Compression [13.161311799049978]
  We propose a predictive coding scheme which uses image animation as a predictor, and codes the residual with respect to the actual target frame. Our experiments indicate a significant gain, in excess of 70% compared to the HEVC video standard and over 30% compared to VVC.
  arXiv  Detail & Related papers  (2023-07-09T14:40:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.