CD-NGP: A Fast Scalable Continual Representation for Dynamic Scenes

• URL: http://arxiv.org/abs/2409.05166v4
• Date: Wed, 23 Oct 2024 02:10:29 GMT
• Title: CD-NGP: A Fast Scalable Continual Representation for Dynamic Scenes
• Authors: Zhenhuan Liu, Shuai Liu, Zhiwei Ning, Jie Yang, Wei Liu,
• Abstract summary: We propose continual dynamic neural graphics primitives (CD-NGP) for view synthesis. Our approach synergizes features from both temporal and spatial hash encodings to achieve high rendering quality. We introduce a novel dataset comprising multi-view, exceptionally long video sequences with substantial rigid and non-rigid motion.
• Score: 9.217592165862762
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Current methodologies for novel view synthesis (NVS) in dynamic scenes encounter significant challenges in harmonizing memory consumption, model complexity, training efficiency, and rendering fidelity. Existing offline techniques, while delivering high-quality results, are often characterized by substantial memory demands and limited scalability. In contrast, online methods grapple with the challenge of balancing rapid convergence with model compactness. To address these issues, we propose continual dynamic neural graphics primitives (CD-NGP). Our approach synergizes features from both temporal and spatial hash encodings to achieve high rendering quality, employs parameter reuse to enhance scalability, and leverages a continual learning framework to mitigate memory overhead. Furthermore, we introduce a novel dataset comprising multi-view, exceptionally long video sequences with substantial rigid and non-rigid motion, thereby substantiating the scalability of our method.

Related papers

• DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer [56.98400572837792]
  DiVE produces high-fidelity, temporally coherent, and cross-view consistent multi-view videos. These innovations collectively achieve a 2.62x speedup with minimal quality degradation.
  arXiv  Detail & Related papers  (2025-04-28T09:20:50Z)
• QuantCache: Adaptive Importance-Guided Quantization with Hierarchical Latent and Layer Caching for Video Generation [84.91431271257437]
  Diffusion Transformers (DiTs) have emerged as a dominant architecture in video generation. DiTs come with significant drawbacks, including increased computational and memory costs. We propose QuantCache, a novel training-free inference acceleration framework.
  arXiv  Detail & Related papers  (2025-03-09T10:31:51Z)
• Token-Efficient Long Video Understanding for Multimodal LLMs [101.70681093383365]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs. We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Raccoon: Multi-stage Diffusion Training with Coarse-to-Fine Curating Videos [15.781862060265519]
  CFC-VIDS-1M is a high-quality video dataset constructed through a systematic coarse-to-fine curation pipeline. We develop RACCOON, a transformer-based architecture with decoupled spatial-temporal attention mechanisms.
  arXiv  Detail & Related papers  (2025-02-28T18:56:35Z)
• Efficient 4D Gaussian Stream with Low Rank Adaptation [8.69899446610606]
  We propose a highly scalable method for dynamic novel view synthesis with continual learning. Our method continuously reconstructs the dynamics with chunks of video frames, reduces the streaming bandwidth by $90%$ while maintaining high rendering quality comparable to the off-line SOTA methods.
  arXiv  Detail & Related papers  (2025-02-23T13:48:52Z)
• Adaptive and Temporally Consistent Gaussian Surfels for Multi-view Dynamic Reconstruction [3.9363268745580426]
  AT-GS is a novel method for reconstructing high-quality dynamic surfaces from multi-view videos through per-frame incremental optimization. We reduce temporal jittering in dynamic surfaces by ensuring consistency in curvature maps across consecutive frames. Our method achieves superior accuracy and temporal coherence in dynamic surface reconstruction, delivering high-fidelity space-time novel view synthesis.
  arXiv  Detail & Related papers  (2024-11-10T21:30:16Z)
• ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation [83.62931466231898]
  This paper presents ARLON, a framework that boosts diffusion Transformers with autoregressive models for long video generation. A latent Vector Quantized Variational Autoencoder (VQ-VAE) compresses the input latent space of the DiT model into compact visual tokens. An adaptive norm-based semantic injection module integrates the coarse discrete visual units from the AR model into the DiT model.
  arXiv  Detail & Related papers  (2024-10-27T16:28:28Z)
• Temporal Feature Matters: A Framework for Diffusion Model Quantization [105.3033493564844]
  Diffusion models rely on the time-step for the multi-round denoising. We introduce a novel quantization framework that includes three strategies. This framework preserves most of the temporal information and ensures high-quality end-to-end generation.
  arXiv  Detail & Related papers  (2024-07-28T17:46:15Z)
• Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design [18.57172631588624]
  We propose a Dynamic Deep neural network assisted by a Content-Aware data processing pipeline to reduce the model number down to one. Our method achieves better PSNR and real-time performance (33 FPS) on an off-the-shelf mobile phone.
  arXiv  Detail & Related papers  (2024-07-03T05:17:26Z)
• RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks [93.18404922542702]
  We present a novel video generative model designed to address long-term spatial and temporal dependencies. Our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks. Our model synthesizes high-fidelity video clips at a resolution of $256times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps.
  arXiv  Detail & Related papers  (2024-01-11T16:48:44Z)
• Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution [65.91317390645163]
  Upscale-A-Video is a text-guided latent diffusion framework for video upscaling. It ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences. It also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation.
  arXiv  Detail & Related papers  (2023-12-11T18:54:52Z)
• Differentiable Resolution Compression and Alignment for Efficient Video Classification and Retrieval [16.497758750494537]
  We propose an efficient video representation network with Differentiable Resolution Compression and Alignment mechanism. We leverage a Differentiable Context-aware Compression Module to encode the saliency and non-saliency frame features. We introduce a new Resolution-Align Transformer Layer to capture global temporal correlations among frame features with different resolutions.
  arXiv  Detail & Related papers  (2023-09-15T05:31:53Z)
• Alignment-free HDR Deghosting with Semantics Consistent Transformer [76.91669741684173]
  High dynamic range imaging aims to retrieve information from multiple low-dynamic range inputs to generate realistic output. Existing methods often focus on the spatial misalignment across input frames caused by the foreground and/or camera motion. We propose a novel alignment-free network with a Semantics Consistent Transformer (SCTNet) with both spatial and channel attention modules.
  arXiv  Detail & Related papers  (2023-05-29T15:03:23Z)
• Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation [55.36617538438858]
  We propose a novel approach that strengthens the interaction between spatial and temporal perceptions. We curate a large-scale and open-source video dataset called HD-VG-130M.
  arXiv  Detail & Related papers  (2023-05-18T11:06:15Z)
• Unbiased Scene Graph Generation in Videos [36.889659781604564]
  We introduce TEMPURA: TEmporal consistency and Memory-guided UnceRtainty Attenuation for unbiased dynamic SGG. TEMPURA employs object-level temporal consistencies via transformer sequence modeling, learns to synthesize unbiased relationship representations. Our method achieves significant (up to 10% in some cases) performance gain over existing methods.
  arXiv  Detail & Related papers  (2023-04-03T06:10:06Z)
• Evolve Smoothly, Fit Consistently: Learning Smooth Latent Dynamics For Advection-Dominated Systems [14.553972457854517]
  We present a data-driven, space-time continuous framework to learn surrogatemodels for complex physical systems. We leverage the expressive power of the network and aspecially designed consistency-inducing regularization to obtain latent trajectories that are both low-dimensional and smooth.
  arXiv  Detail & Related papers  (2023-01-25T03:06:03Z)
• Fast Non-Rigid Radiance Fields from Monocularized Data [66.74229489512683]
  This paper proposes a new method for full 360deg inward-facing novel view synthesis of non-rigidly deforming scenes. At the core of our method are 1) An efficient deformation module that decouples the processing of spatial and temporal information for accelerated training and inference; and 2) A static module representing the canonical scene as a fast hash-encoded neural radiance field. In both cases, our method is significantly faster than previous methods, converging in less than 7 minutes and achieving real-time framerates at 1K resolution, while obtaining a higher visual accuracy for generated novel views.
  arXiv  Detail & Related papers  (2022-12-02T18:51:10Z)
• Gait Recognition in the Wild with Multi-hop Temporal Switch [81.35245014397759]
  gait recognition in the wild is a more practical problem that has attracted the attention of the community of multimedia and computer vision. This paper presents a novel multi-hop temporal switch method to achieve effective temporal modeling of gait patterns in real-world scenes.
  arXiv  Detail & Related papers  (2022-09-01T10:46:09Z)
• Multivariate Time Series Forecasting with Dynamic Graph Neural ODEs [65.18780403244178]
  We propose a continuous model to forecast Multivariate Time series with dynamic Graph neural Ordinary Differential Equations (MTGODE) Specifically, we first abstract multivariate time series into dynamic graphs with time-evolving node features and unknown graph structures. Then, we design and solve a neural ODE to complement missing graph topologies and unify both spatial and temporal message passing.
  arXiv  Detail & Related papers  (2022-02-17T02:17:31Z)
• Temporal-MPI: Enabling Multi-Plane Images for Dynamic Scene Modelling via Temporal Basis Learning [6.952039070065292]
  We propose a novel Temporal-MPI representation which is able to encode the rich 3D and dynamic variation information throughout the entire video as compact temporal basis. Our proposed Temporal-MPI framework is able to generate a time-instance MPI with only 0.002 seconds, which is up to 3000 times faster, with 3dB higher average view-synthesis PSNR as compared with other state-of-the-art dynamic scene modelling frameworks.
  arXiv  Detail & Related papers  (2021-11-20T07:34:28Z)
• Enabling Continual Learning with Differentiable Hebbian Plasticity [18.12749708143404]
  Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. catastrophic forgetting poses a grand challenge for neural networks performing such learning process. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity.
  arXiv  Detail & Related papers  (2020-06-30T06:42:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.