HyperGS: Fast and Generalizable Gaussian Video Representation

• URL: http://arxiv.org/abs/2607.11500v1
• Date: Mon, 13 Jul 2026 12:53:02 GMT
• Title: HyperGS: Fast and Generalizable Gaussian Video Representation
• Abstract summary: HyperGS is a feedforward, optimization-free approach that directly predicts Gaussian representations from any video in a single forward pass.<n>We find that naively predicting Gaussians across diverse videos induces a needle-like degeneration that collapses training.<n>HyperGS improves PSNR by +2.9--3.1 dB over the prior video encoders on K400, SSv2, and UCF101 at a smaller video representation size.
• Score: 50.75993899195029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gaussian Splatting has emerged as an effective representation for video, but existing methods rely on per-video optimization. This leads to slow encoding and limits generalization across videos. To amortize this optimization, we propose HyperGS, a feedforward, optimization-free approach that directly predicts Gaussian representations from any video in a single forward pass, speeding up encoding and decoding by orders of magnitude while generalizing to out-of-distribution videos at higher resolutions. In HyperGS, we design a factorized spatiotemporal Transformer to extract tokens from video, and a learnable query-based Transformer to obtain 8-parameter Gaussian representations for each video frame. We find that naively predicting Gaussians across diverse videos induces a needle-like degeneration that collapses training, and address this with a rank-based geometric regularizer whose strength adapts dynamically to stabilize optimization. HyperGS achieves encoding at $10^4$--$10^5\times$ the speed of per-video Gaussian optimization at matched reconstruction quality while generalizing zero-shot to $720p$ video, enabling higher-resolution rendering without re-encoding. HyperGS improves PSNR by +2.9--3.1 dB over the prior video encoders on K400, SSv2, and UCF101 at a smaller video representation size. By predicting explicit 2D Gaussians in a single forward pass, HyperGS combines the fast, flexible rendering of Gaussian Splatting with the speed and generalization of feedforward prediction, advancing Gaussians as a practical direction for fast and generalizable video representation.

Related papers

• Fast 2DGS: Efficient Image Representation with Deep Gaussian Prior [21.89104780995278]
  Fast-2DGS is a lightweight framework for efficient Gaussian image representation.<n>We introduce Deep Gaussian Prior, implemented as a conditional network to capture the spatial distribution of Gaussian primitives.<n> Experiments demonstrate that this disentangled architecture achieves high-quality reconstruction in a single forward pass, followed by minimal fine-tuning.
  arXiv  Detail & Related papers  (2025-12-14T17:23:28Z)
• Versatile Video Tokenization with Generative 2D Gaussian Splatting [21.242557918885012]
  Video Transformer (GVT) is a versatile video tokenizer built upon a generative 2D Gaussian Splatting strategy.<n>GVT achieves the baseline-of-the-art video quality, outperforms MAGVIT-v2 in action recognition, and delivers comparable compression performance.
  arXiv  Detail & Related papers  (2025-08-15T03:16:45Z)
• GSVR: 2D Gaussian-based Video Representation for 800+ FPS with Hybrid Deformation Field [7.977026024810772]
  Implicit neural representations for video have been recognized as a novel and promising video representation.<n>We propose GSVR, a novel 2D Gaussian-based video representation, which achieves 800+ FPS and 35+ PSNR on Bunny.<n>Our method converges much faster than existing methods and also has 10x faster decoding speed compared to other methods.
  arXiv  Detail & Related papers  (2025-07-08T02:13:12Z)
• Speedy Deformable 3D Gaussian Splatting: Fast Rendering and Compression of Dynamic Scenes [57.69608119350651]
  Recent extensions of 3D Gaussian Splatting (3DGS) to dynamic scenes achieve high-quality novel view synthesis by using neural networks to predict the time-varying deformation of each Gaussian.<n>However, performing per-Gaussian neural inference at every frame poses a significant bottleneck, limiting rendering speed and increasing memory and compute requirements.<n>We present Speedy Deformable 3D Gaussian Splatting (SpeeDe3DGS), a general pipeline for accelerating the rendering speed of dynamic 3DGS and 4DGS representations by reducing neural inference through two complementary techniques.
  arXiv  Detail & Related papers  (2025-06-09T16:30:48Z)
• Second-order Optimization of Gaussian Splats with Importance Sampling [51.95046424364725]
  3D Gaussian Splatting (3DGS) is widely used for novel view rendering due to its high quality and fast inference time.<n>We propose a novel second-order optimization strategy based on Levenberg-Marquardt (LM) and Conjugate Gradient (CG)<n>Our method achieves a $3times$ speedup over standard LM and outperforms Adam by $6times$ when the Gaussian count is low.
  arXiv  Detail & Related papers  (2025-04-17T12:52:08Z)
• GaussianVideo: Efficient Video Representation and Compression by Gaussian Splatting [10.568851068989973]
  Implicit Neural Representation for Videos (NeRV) has introduced a novel paradigm for video representation and compression.<n>We propose a new video representation and method based on 2D Gaussian Splatting to efficiently handle data handle.<n>Our method reduces memory usage by up to 78.4%, and significantly expedites video processing, achieving 5.5x faster training and 12.5x faster decoding.
  arXiv  Detail & Related papers  (2025-03-06T11:31:08Z)
• GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting [3.479384894190067]
  We propose GSVC, an approach to learning a set of 2D Gaussian splats that can effectively represent and compress video frames.<n>Experiment results show that GSVC achieves good rate-distortion trade-offs, comparable to state-of-the-art video codecs.
  arXiv  Detail & Related papers  (2025-01-21T11:30:51Z)
• VeGaS: Video Gaussian Splatting [0.42881773214459123]
  We introduce the Video Gaussian Splatting (VeGaS) model, which enables realistic modifications of video data. VeGaS outperforms state-of-the-art solutions in frame reconstruction tasks.
  arXiv  Detail & Related papers  (2024-11-17T10:02:36Z)
• MEGA: Memory-Efficient 4D Gaussian Splatting for Dynamic Scenes [49.36091070642661]
  This paper introduces a memory-efficient framework for 4DGS.<n>It achieves a storage reduction by approximately 190$times$ and 125$times$ on the Technicolor and Neural 3D Video datasets.<n>It maintains comparable rendering speeds and scene representation quality, setting a new standard in the field.
  arXiv  Detail & Related papers  (2024-10-17T14:47:08Z)
• 3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt [65.25603275491544]
  We present 3DGS-LM, a new method that accelerates the reconstruction of 3D Gaussian Splatting (3DGS)<n>Our method is 20% faster than the original 3DGS while obtaining the same reconstruction quality optimization.
  arXiv  Detail & Related papers  (2024-09-19T16:31:44Z)
• MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo [54.00987996368157]
  We present MVSGaussian, a new generalizable 3D Gaussian representation approach derived from Multi-View Stereo (MVS) MVSGaussian achieves real-time rendering with better synthesis quality for each scene.
  arXiv  Detail & Related papers  (2024-05-20T17:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.