Efficient Event Stream Super-Resolution with Recursive Multi-Branch Fusion

• URL: http://arxiv.org/abs/2406.19640v1
• Date: Fri, 28 Jun 2024 04:10:21 GMT
• Title: Efficient Event Stream Super-Resolution with Recursive Multi-Branch Fusion
• Authors: Quanmin Liang, Zhilin Huang, Xiawu Zheng, Feidiao Yang, Jun Peng, Kai Huang, Yonghong Tian,
• Abstract summary: We propose an efficient Recursive Multi-Branch Information Fusion Network (RMFNet) to separate positive and negative events. FEM efficiently promotes the fusion and exchange of information between positive and negative branches. Our approach achieves over 17% and 31% improvement on synthetic and real datasets, accompanied by a 2.3X acceleration.
• Score: 30.746523517295007
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current Event Stream Super-Resolution (ESR) methods overlook the redundant and complementary information present in positive and negative events within the event stream, employing a direct mixing approach for super-resolution, which may lead to detail loss and inefficiency. To address these issues, we propose an efficient Recursive Multi-Branch Information Fusion Network (RMFNet) that separates positive and negative events for complementary information extraction, followed by mutual supplementation and refinement. Particularly, we introduce Feature Fusion Modules (FFM) and Feature Exchange Modules (FEM). FFM is designed for the fusion of contextual information within neighboring event streams, leveraging the coupling relationship between positive and negative events to alleviate the misleading of noises in the respective branches. FEM efficiently promotes the fusion and exchange of information between positive and negative branches, enabling superior local information enhancement and global information complementation. Experimental results demonstrate that our approach achieves over 17% and 31% improvement on synthetic and real datasets, accompanied by a 2.3X acceleration. Furthermore, we evaluate our method on two downstream event-driven applications, \emph{i.e.}, object recognition and video reconstruction, achieving remarkable results that outperform existing methods. Our code and Supplementary Material are available at https://github.com/Lqm26/RMFNet.

Related papers

• LASFNet: A Lightweight Attention-Guided Self-Modulation Feature Fusion Network for Multimodal Object Detection [4.2649265429416445]
  We propose a new fusion detection baseline that uses a single feature-level fusion unit to enable high-performance detection.<n>Based on this approach, we propose a lightweight attention-guided self-modulation feature fusion network (LASFNet)<n>Our approach achieves a favorable efficiency-accuracy trade-off, reducing the number of parameters and computational cost by as much as 90% and 85%, respectively.
  arXiv  Detail & Related papers  (2025-06-26T05:32:33Z)
• Spatially-guided Temporal Aggregation for Robust Event-RGB Optical Flow Estimation [47.75348821902489]
  Current optical flow methods exploit the stable appearance of frame (or RGB) data to establish robust correspondences across time. Event cameras, on the other hand, provide high-temporal-resolution motion cues and excel in challenging scenarios. This study introduces a novel approach that uses a spatially dense modality to guide the aggregation of the temporally dense event modality.
  arXiv  Detail & Related papers  (2025-01-01T13:40:09Z)
• Rethinking Normalization Strategies and Convolutional Kernels for Multimodal Image Fusion [25.140475569677758]
  Multimodal image fusion aims to integrate information from different modalities to obtain a comprehensive image. Existing methods tend to prioritize natural image fusion and focus on information complementary and network training strategies. This paper dissects the significant differences between the two tasks regarding fusion goals, statistical properties, and data distribution.
  arXiv  Detail & Related papers  (2024-11-15T08:36:24Z)
• Bilateral Event Mining and Complementary for Event Stream Super-Resolution [28.254644673666903]
  Event Stream Super-Resolution (ESR) aims to address the challenge of insufficient spatial resolution in event streams. We propose a bilateral event mining and complementary network (BMCNet) to fully leverage the potential of each event. Our method significantly enhances the performance of event-based downstream tasks such as object recognition and video reconstruction.
  arXiv  Detail & Related papers  (2024-05-16T12:16:25Z)
• Feature Decoupling-Recycling Network for Fast Interactive Segmentation [79.22497777645806]
  Recent interactive segmentation methods iteratively take source image, user guidance and previously predicted mask as the input. We propose the Feature Decoupling-Recycling Network (FDRN), which decouples the modeling components based on their intrinsic discrepancies.
  arXiv  Detail & Related papers  (2023-08-07T12:26:34Z)
• FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration [27.14686854704104]
  We propose a Feature Fusion two-stream framework (FF2) for punctuation restoration. Specifically, one stream leverages a pre-trained language model to capture the semantic feature, while another auxiliary module captures the feature at hand. Without additional data, the experimental results on the popular benchmark IWSLT demonstrate that FF2 achieves new SOTA performance.
  arXiv  Detail & Related papers  (2022-11-09T06:18:17Z)
• Magic ELF: Image Deraining Meets Association Learning and Transformer [63.761812092934576]
  This paper aims to unify CNN and Transformer to take advantage of their learning merits for image deraining. A novel multi-input attention module (MAM) is proposed to associate rain removal and background recovery. Our proposed method (dubbed as ELF) outperforms the state-of-the-art approach (MPRNet) by 0.25 dB on average.
  arXiv  Detail & Related papers  (2022-07-21T12:50:54Z)
• Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection [77.50110439560152]
  Current object detectors typically have a feature pyramid (FP) module for multi-level feature fusion (MFF) We propose a novel and efficient context modeling mechanism that can help existing FPs deliver better MFF results. In particular, we introduce a novel insight that comprehensive contexts can be decomposed and condensed into two types of representations for higher efficiency.
  arXiv  Detail & Related papers  (2022-07-14T01:45:03Z)
• Decoupled Side Information Fusion for Sequential Recommendation [6.515279047538104]
  We propose Decoupled Side Information Fusion for Sequential Recommendation (DIF-SR) It moves the side information from the input to the attention layer and decouples the attention calculation of various side information and item representation. Our proposed solution stably outperforms state-of-the-art SR models.
  arXiv  Detail & Related papers  (2022-04-23T10:53:36Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation [62.210091681352914]
  We study multi-sensor fusion for 3D semantic segmentation for many applications, such as autonomous driving and robotics. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) We propose a two-stream network to extract features from the two modalities separately. The extracted features are fused by effective residual-based fusion modules.
  arXiv  Detail & Related papers  (2021-06-21T10:47:26Z)
• Deep Multimodal Fusion by Channel Exchanging [87.40768169300898]
  This paper proposes a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network.
  arXiv  Detail & Related papers  (2020-11-10T09:53:20Z)
• Dual Semantic Fusion Network for Video Object Detection [35.175552056938635]
  We propose a dual semantic fusion network (DSFNet) to fully exploit both frame-level and instance-level semantics in a unified fusion framework without external guidance. The proposed DSFNet can generate more robust features through the multi-granularity fusion and avoid being affected by the instability of external guidance.
  arXiv  Detail & Related papers  (2020-09-16T06:49:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.