ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression

• URL: http://arxiv.org/abs/2607.28020v1
• Date: Thu, 30 Jul 2026 11:10:13 GMT
• Title: ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression
• Abstract summary: Event-Assisted Complementary Motion Refinement framework for learned video compression.<n> ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings.<n> experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths.
• Score: 16.49973800168176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learned video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two datasets.

Related papers

• RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics [93.19893734404417]
  Video deraining aims to recover clean visual content from rainy videos for reliable perception under adverse weather.<n>Existing methods mainly rely on RGB sequences and temporal redundancy, but RGB-only restoration remains ambiguous in dynamic rainy scenes.<n>We propose RainDancer, a progressive RGB-Event video deraining framework based on a decompose-before-interact paradigm.
  arXiv  Detail & Related papers  (2026-07-15T13:08:21Z)
• Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing [55.13802890769086]
  We present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing.<n>To achieve profound modal synergy, Evita embeds a suite of intrinsic co-learning modules directly into every encoder layer.<n>Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.
  arXiv  Detail & Related papers  (2026-07-10T06:52:09Z)
• UniRED: Unified RGB-D Video Frame Interpolation with Event Guidance [19.80142530142947]
  High frame-rate RGB-D videos are crucial for a variety of downstream tasks, including motion analysis, dynamic scene understanding, and 3D reconstruction.<n>Existing video methods have achieved strong performance on RGB data, but they are not readily applicable to RGB-D scenarios.<n>We propose a unified multimodal framework for RGB-D video that jointly exploits RGB appearance, depth geometry, and event-based temporal cues.
  arXiv  Detail & Related papers  (2026-06-23T08:03:58Z)
• Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor [12.418360718687026]
  Motion blur arises when rapid scene changes occur during the exposure period, collapsing rich intra-exposure motion into a single RGB frame.<n>We propose Spatio-Temporal Difference Guided Deblur Net (STGDNet) to restore structure and color details lost in blurry RGB inputs.<n>Our method outperforms current RGB or event-based approaches in both synthetic CVS dataset and real-world evaluations.
  arXiv  Detail & Related papers  (2026-04-12T09:54:14Z)
• Spatial Orthogonal Refinement for Robust RGB-Event Visual Object Tracking [8.019596736149834]
  We propose SOR-Track, a framework for robust RGB-Event tracking based on Spatial Orthogonal Refinement (SOR)<n>Experiments on the large-scale FE108 benchmark demonstrate that SOR-Track consistently outperforms existing fusion-based trackers.<n>Despite its simplicity, the proposed method offers a principled and physics-grounded approach to multi-modal feature alignment and texture rectification.
  arXiv  Detail & Related papers  (2026-03-29T23:54:31Z)
• Color When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensing [58.25947399492333]
  We present a new paradigm for efficient streaming video understanding: grayscale-always, color-on-demand.<n>We propose ColorTrigger, an online training-free trigger that activates color capture based on windowed grayscale affinity analysis.<n>On streaming video understanding benchmarks, ColorTrigger achieves 91.6% of full-color baseline performance while using only 8.1% RGB frames.
  arXiv  Detail & Related papers  (2026-03-23T18:32:36Z)
• Decoupling Amplitude and Phase Attention in Frequency Domain for RGB-Event based Visual Object Tracking [51.31378940976401]
  Existing RGB-Event tracking approaches fail to fully exploit the unique advantages of event cameras.<n>We propose a novel tracking framework that performs early fusion in the frequency domain, enabling effective aggregation of high-frequency information from the event modality.<n>Experiments on three widely used RGB-Event tracking benchmark datasets, including FE108, FELT, and COESOT, demonstrate the high performance and efficiency of our method.
  arXiv  Detail & Related papers  (2026-01-03T01:10:17Z)
• Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking [74.15663758681849]
  We present Learning Frequency and Memory-Aware Prompts, a dual-adapter framework that injects lightweight prompts into a frozen RGB tracker.<n>A frequency-guided visual adapter adaptively transfers complementary cues across modalities.<n>A multilevel memory adapter with short, long, and permanent memory stores, updates, and retrieves reliable temporal context.
  arXiv  Detail & Related papers  (2025-06-30T15:38:26Z)
• End-to-End RGB-IR Joint Image Compression With Channel-wise Cross-modality Entropy Model [39.52468600966148]
  As the number of modalities increases, the required data storage and transmission costs also double.<n>This work proposes a joint compression framework for RGB-IR image pair.
  arXiv  Detail & Related papers  (2025-06-27T02:04:21Z)
• Learning Flow-Guided Registration for RGB-Event Semantic Segmentation [22.996619370156584]
  Event cameras capture microsecond-level motion cues that complement RGB sensors.<n>We recast RGB-Event segmentation from fusion to registration.<n>We propose BRENet, a novel flow-guided bidirectional framework that adaptively matches correspondence between the asymmetric modalities.
  arXiv  Detail & Related papers  (2025-05-02T19:19:58Z)
• Frequency-Adaptive Low-Latency Object Detection Using Events and Frames [23.786369609995013]
  Fusing Events and RGB images for object detection leverages the robustness of Event cameras in adverse environments.<n>Two critical mismatches: low-latency Events textitvs.high-latency RGB frames, and temporally sparse labels in training textitvs.continuous flow in inference.<n>We propose the textbfFrequency-textbfAdaptive Low-Latency textbfObject textbfDetector (FAOD)
  arXiv  Detail & Related papers  (2024-12-05T13:23:06Z)
• Revisiting Event-based Video Frame Interpolation [49.27404719898305]
  Dynamic vision sensors or event cameras provide rich complementary information for video frame. estimating optical flow from events is arguably more difficult than from RGB information. We propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages.
  arXiv  Detail & Related papers  (2023-07-24T06:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.