Fine-Grained Motion Compression and Selective Temporal Fusion for Neural B-Frame Video Coding

• URL: http://arxiv.org/abs/2506.07709v1
• Date: Mon, 09 Jun 2025 12:51:10 GMT
• Title: Fine-Grained Motion Compression and Selective Temporal Fusion for Neural B-Frame Video Coding
• Authors: Xihua Sheng, Peilin Chen, Meng Wang, Li Zhang, Shiqi Wang, Dapeng Oliver Wu,
• Abstract summary: We propose novel enhancements for motion compression and temporal fusion for neural B-frame coding.<n>Our proposed method incorporates an interactive dual-branch motion auto-encoder with per-branch adaptive quantization steps.<n>Second, we propose a selective temporal fusion method that predicts bi-directional fusion weights to achieve discriminative utilization of bi-directional multi-scale temporal contexts.
• Score: 27.315485948158006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the remarkable progress in neural P-frame video coding, neural B-frame coding has recently emerged as a critical research direction. However, most existing neural B-frame codecs directly adopt P-frame coding tools without adequately addressing the unique challenges of B-frame compression, leading to suboptimal performance. To bridge this gap, we propose novel enhancements for motion compression and temporal fusion for neural B-frame coding. First, we design a fine-grained motion compression method. This method incorporates an interactive dual-branch motion auto-encoder with per-branch adaptive quantization steps, which enables fine-grained compression of bi-directional motion vectors while accommodating their asymmetric bitrate allocation and reconstruction quality requirements. Furthermore, this method involves an interactive motion entropy model that exploits correlations between bi-directional motion latent representations by interactively leveraging partitioned latent segments as directional priors. Second, we propose a selective temporal fusion method that predicts bi-directional fusion weights to achieve discriminative utilization of bi-directional multi-scale temporal contexts with varying qualities. Additionally, this method introduces a hyperprior-based implicit alignment mechanism for contextual entropy modeling. By treating the hyperprior as a surrogate for the contextual latent representation, this mechanism implicitly mitigates the misalignment in the fused bi-directional temporal priors. Extensive experiments demonstrate that our proposed codec outperforms state-of-the-art neural B-frame codecs and achieves comparable or even superior compression performance to the H.266/VVC reference software under random-access configurations.

Related papers

• Bi-Directional Deep Contextual Video Compression [17.195099321371526]
  We introduce a bi-directional deep contextual video compression scheme tailored for B-frames, termed DCVC-B.<n>First, we develop a bi-directional motion difference context propagation method for effective motion difference coding.<n>Second, we propose a bi-directional contextual compression model and a corresponding bi-directional temporal entropy model.<n>Third, we propose a hierarchical quality structure-based training strategy, leading to an effective bit allocation across large groups of pictures.
  arXiv  Detail & Related papers  (2024-08-16T08:45:25Z)
• IBVC: Interpolation-driven B-frame Video Compression [68.18440522300536]
  B-frame video compression aims to adopt bi-directional motion estimation and motion compensation (MEMC) coding for middle frame reconstruction. Previous learned approaches often directly extend neural P-frame codecs to B-frame relying on bi-directional optical-flow estimation. We propose a simple yet effective structure called Interpolation-B-frame Video Compression (IBVC) to address these issues.
  arXiv  Detail & Related papers  (2023-09-25T02:45:51Z)
• Spatial-Temporal Transformer based Video Compression Framework [44.723459144708286]
  We propose a novel Spatial-Temporal Transformer based Video Compression (STT-VC) framework. It contains a Relaxed Deformable Transformer (RDT) with Uformer based offsets estimation for motion estimation and compensation, a Multi-Granularity Prediction (MGP) module based on multi-reference frames for prediction refinement, and a Spatial Feature Distribution prior based Transformer (SFD-T) for efficient temporal-spatial joint residual compression. Experimental results demonstrate that our method achieves the best result with 13.5% BD-Rate saving over VTM.
  arXiv  Detail & Related papers  (2023-09-21T09:23:13Z)
• Dynamic Frame Interpolation in Wavelet Domain [57.25341639095404]
  Video frame is an important low-level computation vision task, which can increase frame rate for more fluent visual experience. Existing methods have achieved great success by employing advanced motion models and synthesis networks. WaveletVFI can reduce computation up to 40% while maintaining similar accuracy, making it perform more efficiently against other state-of-the-arts.
  arXiv  Detail & Related papers  (2023-09-07T06:41:15Z)
• Modality-Agnostic Variational Compression of Implicit Neural Representations [96.35492043867104]
  We introduce a modality-agnostic neural compression algorithm based on a functional view of data and parameterised as an Implicit Neural Representation (INR) Bridging the gap between latent coding and sparsity, we obtain compact latent representations non-linearly mapped to a soft gating mechanism. After obtaining a dataset of such latent representations, we directly optimise the rate/distortion trade-off in a modality-agnostic space using neural compression.
  arXiv  Detail & Related papers  (2023-01-23T15:22:42Z)
• Learned Video Compression via Heterogeneous Deformable Compensation Network [78.72508633457392]
  We propose a learned video compression framework via heterogeneous deformable compensation strategy (HDCVC) to tackle the problems of unstable compression performance. More specifically, the proposed algorithm extracts features from the two adjacent frames to estimate content-Neighborhood heterogeneous deformable (HetDeform) kernel offsets. Experimental results indicate that HDCVC achieves superior performance than the recent state-of-the-art learned video compression approaches.
  arXiv  Detail & Related papers  (2022-07-11T02:31:31Z)
• End-to-end Neural Video Coding Using a Compound Spatiotemporal Representation [33.54844063875569]
  We propose a hybrid motion compensation (HMC) method that adaptively combines the predictions generated by two approaches. Specifically, we generate a compoundtemporal representation (STR) through a recurrent information aggregation (RIA) module. We further design a one-to-many decoder pipeline to generate multiple predictions from the CSTR, including vector-based resampling, adaptive kernel-based resampling, compensation mode selection maps and texture enhancements.
  arXiv  Detail & Related papers  (2021-08-05T19:43:32Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling [52.425236515695914]
  State-of-the-art methods are iterative solutions interpolating one frame at the time. This work introduces a true multi-frame interpolator. It utilizes a pyramidal style network in the temporal domain to complete the multi-frame task in one-shot.
  arXiv  Detail & Related papers  (2020-07-23T02:34:39Z)
• Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model [45.46660511313426]
  We propose an end-to-end deep neural video coding framework (NVC) It uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods.
  arXiv  Detail & Related papers  (2020-07-09T06:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.