Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition

• URL: http://arxiv.org/abs/2406.18845v1
• Date: Thu, 27 Jun 2024 02:32:46 GMT
• Title: Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition
• Authors: Lan Chen, Dong Li, Xiao Wang, Pengpeng Shao, Wei Zhang, Yaowei Wang, Yonghong Tian, Jin Tang,
• Abstract summary: We propose a novel dual-stream framework for event stream-based pattern recognition via differentiated fusion, termed EFV++. It models two common event representations simultaneously, i.e., event images and event voxels. We achieve new state-of-the-art performance on the Bullying10k dataset, i.e., $90.51%$, which exceeds the second place by $+2.21%$.
• Score: 57.74076383449153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing event stream-based pattern recognition models usually represent the event stream as the point cloud, voxel, image, etc., and design various deep neural networks to learn their features. Although considerable results can be achieved in simple cases, however, the model performance may be limited by monotonous modality expressions, sub-optimal fusion, and readout mechanisms. In this paper, we propose a novel dual-stream framework for event stream-based pattern recognition via differentiated fusion, termed EFV++. It models two common event representations simultaneously, i.e., event images and event voxels. The spatial and three-dimensional stereo information can be learned separately by utilizing Transformer and Graph Neural Network (GNN). We believe the features of each representation still contain both efficient and redundant features and a sub-optimal solution may be obtained if we directly fuse them without differentiation. Thus, we divide each feature into three levels and retain high-quality features, blend medium-quality features, and exchange low-quality features. The enhanced dual features will be fed into the fusion Transformer together with bottleneck features. In addition, we introduce a novel hybrid interaction readout mechanism to enhance the diversity of features as final representations. Extensive experiments demonstrate that our proposed framework achieves state-of-the-art performance on multiple widely used event stream-based classification datasets. Specifically, we achieve new state-of-the-art performance on the Bullying10k dataset, i.e., $90.51\%$, which exceeds the second place by $+2.21\%$. The source code of this paper has been released on \url{https://github.com/Event-AHU/EFV_event_classification/tree/EFVpp}.

Related papers

• Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classification [6.550582412924754]
  This paper proposes a novel dual-stream framework for event representation, extraction, and fusion. Experiments demonstrate that our proposed framework achieves state-of-the-art performance on two widely used event-based classification datasets.
  arXiv  Detail & Related papers  (2023-08-23T06:07:56Z)
• Feature Decoupling-Recycling Network for Fast Interactive Segmentation [79.22497777645806]
  Recent interactive segmentation methods iteratively take source image, user guidance and previously predicted mask as the input. We propose the Feature Decoupling-Recycling Network (FDRN), which decouples the modeling components based on their intrinsic discrepancies.
  arXiv  Detail & Related papers  (2023-08-07T12:26:34Z)
• Multi-scale Alternated Attention Transformer for Generalized Stereo Matching [7.493797166406228]
  We present a simple but highly effective network called Alternated Attention U-shaped Transformer (AAUformer) to balance the impact of epipolar line in dual and single view. Compared to other models, our model has several main designs. We performed a series of both comparative studies and ablation studies on several mainstream stereo matching datasets.
  arXiv  Detail & Related papers  (2023-08-06T08:22:39Z)
• A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence [83.90531416914884]
  We exploit Stable Diffusion features for semantic and dense correspondence. With simple post-processing, SD features can perform quantitatively similar to SOTA representations. We show that these correspondences can enable interesting applications such as instance swapping in two images.
  arXiv  Detail & Related papers  (2023-05-24T16:59:26Z)
• Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence [88.00004819064672]
  Diffusion Hyperfeatures is a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors. Our method achieves superior performance on the SPair-71k real image benchmark.
  arXiv  Detail & Related papers  (2023-05-23T17:58:05Z)
• Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams [23.872611710730865]
  Event cameras are neuromorphic vision sensors representing visual information as sparse and asynchronous event streams. We develop a novel attention-aware model named Event Voxel Set Transformer (EVSTr) for representation learning on event streams. We evaluate the proposed model on two event-based recognition tasks: object classification and action recognition.
  arXiv  Detail & Related papers  (2023-03-07T12:48:02Z)
• HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities [6.543272301133159]
  Event cameras detect changes in per-pixel intensity to generate asynchronous event streams. They offer great potential for accurate semantic map retrieval in real-time autonomous systems. Existing implementations for event segmentation suffer from sub-based performance. We propose hybrid end-to-end learning framework HALSIE to reduce inference cost by up to $20times$ versus art.
  arXiv  Detail & Related papers  (2022-11-19T17:09:50Z)
• Unifying Flow, Stereo and Depth Estimation [121.54066319299261]
  We present a unified formulation and model for three motion and 3D perception tasks. We formulate all three tasks as a unified dense correspondence matching problem. Our model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks.
  arXiv  Detail & Related papers  (2022-11-10T18:59:54Z)
• Multi-Scale Hourglass Hierarchical Fusion Network for Single Image Deraining [8.964751500091005]
  Rain streaks bring serious blurring and visual quality degradation, which often vary in size, direction and density. Current CNN-based methods achieve encouraging performance, while are limited to depict rain characteristics and recover image details in the poor visibility environment. We present a Multi-scale Hourglass Hierarchical Fusion Network (MH2F-Net) in end-to-end manner, to exactly captures rain streak features with multi-scale extraction, hierarchical distillation and information aggregation.
  arXiv  Detail & Related papers  (2021-04-25T08:27:01Z)
• Adaptive Context-Aware Multi-Modal Network for Depth Completion [107.15344488719322]
  We propose to adopt the graph propagation to capture the observed spatial contexts. We then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks.
  arXiv  Detail & Related papers  (2020-08-25T06:00:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.