Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach

• URL: http://arxiv.org/abs/2507.03765v1
• Date: Fri, 04 Jul 2025 18:30:28 GMT
• Title: Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach
• Authors: Hebei Li, Yansong Peng, Jiahui Yuan, Peixi Wu, Jin Wang, Yueyi Zhang, Xiaoyan Sun,
• Abstract summary: Event cameras have recently been introduced into image semantic segmentation, owing to their high temporal resolution and other advantageous properties.<n>We propose an efficient hybrid framework for image semantic segmentation, comprising a Spiking Network branch for events and an Artificial Neural Network branch for frames.<n>Specifically, we introduce three specialized modules to facilitate the interaction between these two branches: the Adaptive Temporal Weighting (ATW), the Event-Driven Sparse (EDS), and the Channel Selection Fusion (CSF) module.
• Score: 13.681130802391223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Event cameras have recently been introduced into image semantic segmentation, owing to their high temporal resolution and other advantageous properties. However, existing event-based semantic segmentation methods often fail to fully exploit the complementary information provided by frames and events, resulting in complex training strategies and increased computational costs. To address these challenges, we propose an efficient hybrid framework for image semantic segmentation, comprising a Spiking Neural Network branch for events and an Artificial Neural Network branch for frames. Specifically, we introduce three specialized modules to facilitate the interaction between these two branches: the Adaptive Temporal Weighting (ATW) Injector, the Event-Driven Sparse (EDS) Injector, and the Channel Selection Fusion (CSF) module. The ATW Injector dynamically integrates temporal features from event data into frame features, enhancing segmentation accuracy by leveraging critical dynamic temporal information. The EDS Injector effectively combines sparse event data with rich frame features, ensuring precise temporal and spatial information alignment. The CSF module selectively merges these features to optimize segmentation performance. Experimental results demonstrate that our framework not only achieves state-of-the-art accuracy across the DDD17-Seg, DSEC-Semantic, and M3ED-Semantic datasets but also significantly reduces energy consumption, achieving a 65\% reduction on the DSEC-Semantic dataset.

Related papers

• EIFNet: Leveraging Event-Image Fusion for Robust Semantic Segmentation [0.18416014644193066]
  Event cameras offer high dynamic range and fine temporal resolution, to achieve robust scene understanding in challenging environments.<n>We propose EIFNet, a multi-modal fusion network that combines the strengths of both event and frame-based inputs.<n>EIFNet achieves state-of-the-art performance, demonstrating its effectiveness in event-based semantic segmentation.
  arXiv  Detail & Related papers  (2025-07-29T16:19:55Z)
• ContextFormer: Redefining Efficiency in Semantic Segmentation [48.81126061219231]
  Convolutional methods, although capturing local dependencies well, struggle with long-range relationships.<n>Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands.<n>We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation.
  arXiv  Detail & Related papers  (2025-01-31T16:11:04Z)
• Spatially-guided Temporal Aggregation for Robust Event-RGB Optical Flow Estimation [47.75348821902489]
  Current optical flow methods exploit the stable appearance of frame (or RGB) data to establish robust correspondences across time.<n>Event cameras, on the other hand, provide high-temporal-resolution motion cues and excel in challenging scenarios.<n>This study introduces a novel approach that uses a spatially dense modality to guide the aggregation of the temporally dense event modality.
  arXiv  Detail & Related papers  (2025-01-01T13:40:09Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation [79.02808071245634]
  Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars.
  arXiv  Detail & Related papers  (2023-03-17T12:12:41Z)
• HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalities [6.543272301133159]
  Event cameras detect changes in per-pixel intensity to generate asynchronous event streams. They offer great potential for accurate semantic map retrieval in real-time autonomous systems. Existing implementations for event segmentation suffer from sub-based performance. We propose hybrid end-to-end learning framework HALSIE to reduce inference cost by up to $20times$ versus art.
  arXiv  Detail & Related papers  (2022-11-19T17:09:50Z)
• Ret3D: Rethinking Object Relations for Efficient 3D Object Detection in Driving Scenes [82.4186966781934]
  We introduce a simple, efficient, and effective two-stage detector, termed as Ret3D. At the core of Ret3D is the utilization of novel intra-frame and inter-frame relation modules. With negligible extra overhead, Ret3D achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-08-18T03:48:58Z)
• LiDAR-based Recurrent 3D Semantic Segmentation with Temporal Memory Alignment [0.0]
  We propose a recurrent segmentation architecture (RNN), which takes a single range image frame as input. An alignment strategy, which we call Temporal Memory Alignment, uses ego motion to temporally align the memory between consecutive frames in feature space. We demonstrate the benefits of the presented approach on two large-scale datasets and compare it to several stateof-the-art methods.
  arXiv  Detail & Related papers  (2021-03-03T09:01:45Z)
• Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes [98.65457534223539]
  We propose a real-time high-performance DCNN-based method for robust semantic segmentation of urban street scenes. The proposed method achieves the accuracy of 73.6% and 68.0% mean Intersection over Union (mIoU) with the inference speed of 51.0 fps and 39.3 fps.
  arXiv  Detail & Related papers  (2020-03-11T08:45:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.