Event-Free Moving Object Segmentation from Moving Ego Vehicle
- URL: http://arxiv.org/abs/2305.00126v2
- Date: Tue, 28 Nov 2023 13:12:39 GMT
- Title: Event-Free Moving Object Segmentation from Moving Ego Vehicle
- Authors: Zhuyun Zhou, Zongwei Wu, Danda Pani Paudel, R\'emi Boutteau, Fan Yang,
Luc Van Gool, Radu Timofte, Dominique Ginhac
- Abstract summary: Moving object segmentation (MOS) in dynamic scenes is challenging for autonomous driving.
Most state-of-the-art methods leverage motion cues obtained from optical flow maps.
We propose to exploit event cameras for better video understanding, which provide rich motion cues without relying on optical flow.
- Score: 90.66285408745453
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Moving object segmentation (MOS) in dynamic scenes is challenging for
autonomous driving, especially for sequences obtained from moving ego vehicles.
Most state-of-the-art methods leverage motion cues obtained from optical flow
maps. However, since these methods are often based on optical flows that are
pre-computed from successive RGB frames, this neglects the temporal
consideration of events occurring within inter-frame and limits the
practicality of these methods in real-life situations. To address these
limitations, we propose to exploit event cameras for better video
understanding, which provide rich motion cues without relying on optical flow.
To foster research in this area, we first introduce a novel large-scale dataset
called DSEC-MOS for moving object segmentation from moving ego vehicles.
Subsequently, we devise EmoFormer, a novel network able to exploit the event
data. For this purpose, we fuse the event prior with spatial semantic maps to
distinguish moving objects from the static background, adding another level of
dense supervision around our object of interest - moving ones. Our proposed
network relies only on event data for training but does not require event input
during inference, making it directly comparable to frame-only methods in terms
of efficiency and more widely usable in many application cases. An exhaustive
comparison with 8 state-of-the-art video object segmentation methods highlights
a significant performance improvement of our method over all other methods.
Project Page: https://github.com/ZZY-Zhou/DSEC-MOS.
Related papers
- StreamMOS: Streaming Moving Object Segmentation with Multi-View Perception and Dual-Span Memory [21.300636683882338]
We propose a streaming network with a memory mechanism, called StreamMOS, to build the association of features and predictions among multiple inferences.
Specifically, we utilize a short-term memory to convey historical features, which can be regarded as spatial prior to moving objects.
We also present multi-view encoder with projection and asymmetric convolution to extract motion feature of objects in different representations.
arXiv Detail & Related papers (2024-07-25T09:51:09Z) - Motion Segmentation for Neuromorphic Aerial Surveillance [42.04157319642197]
Event cameras offer superior temporal resolution, superior dynamic range, and minimal power requirements.
Unlike traditional frame-based sensors that capture redundant information at fixed intervals, event cameras asynchronously record pixel-level brightness changes.
We introduce a novel motion segmentation method that leverages self-supervised vision transformers on both event data and optical flow information.
arXiv Detail & Related papers (2024-05-24T04:36:13Z) - Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation
for Complex Scenes [10.936350433952668]
Rapid and reliable identification of dynamic scene parts, also known as motion segmentation, is a key challenge for mobile sensors.
Event cameras have the potential to overcome these limitations, but corresponding methods have only been demonstrated in smaller-scale indoor environments.
This work presents an event-based method for class-agnostic motion segmentation that can successfully be deployed across complex large-scale outdoor environments too.
arXiv Detail & Related papers (2024-03-07T14:59:34Z) - Appearance-Based Refinement for Object-Centric Motion Segmentation [85.2426540999329]
We introduce an appearance-based refinement method that leverages temporal consistency in video streams to correct inaccurate flow-based proposals.
Our approach involves a sequence-level selection mechanism that identifies accurate flow-predicted masks as exemplars.
Its performance is evaluated on multiple video segmentation benchmarks, including DAVIS, YouTube, SegTrackv2, and FBMS-59.
arXiv Detail & Related papers (2023-12-18T18:59:51Z) - SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features [52.213656737672935]
SpikeMOT is an event-based multi-object tracker.
SpikeMOT uses spiking neural networks to extract sparsetemporal features from event streams associated with objects.
arXiv Detail & Related papers (2023-09-29T05:13:43Z) - InstMove: Instance Motion for Object-centric Video Segmentation [70.16915119724757]
In this work, we study the instance-level motion and present InstMove, which stands for Instance Motion for Object-centric Video.
In comparison to pixel-wise motion, InstMove mainly relies on instance-level motion information that is free from image feature embeddings.
With only a few lines of code, InstMove can be integrated into current SOTA methods for three different video segmentation tasks.
arXiv Detail & Related papers (2023-03-14T17:58:44Z) - InsMOS: Instance-Aware Moving Object Segmentation in LiDAR Data [13.196031553445117]
We propose a novel network that addresses the challenge of segmenting moving objects in 3D LiDAR scans.
Our method exploits a sequence of point clouds as input and quantifies them into 4D voxels.
We use 4D sparse convolutions to extract motion features from the 4D voxels and inject them into the current scan.
arXiv Detail & Related papers (2023-03-07T14:12:52Z) - EAN: Event Adaptive Network for Enhanced Action Recognition [66.81780707955852]
We propose a unified action recognition framework to investigate the dynamic nature of video content.
First, when extracting local cues, we generate the spatial-temporal kernels of dynamic-scale to adaptively fit the diverse events.
Second, to accurately aggregate these cues into a global video representation, we propose to mine the interactions only among a few selected foreground objects by a Transformer.
arXiv Detail & Related papers (2021-07-22T15:57:18Z) - DS-Net: Dynamic Spatiotemporal Network for Video Salient Object
Detection [78.04869214450963]
We propose a novel dynamic temporal-temporal network (DSNet) for more effective fusion of temporal and spatial information.
We show that the proposed method achieves superior performance than state-of-the-art algorithms.
arXiv Detail & Related papers (2020-12-09T06:42:30Z) - Self-supervised Sparse to Dense Motion Segmentation [13.888344214818737]
We propose a self supervised method to learn the densification of sparse motion segmentations from single video frames.
We evaluate our method on the well-known motion segmentation datasets FBMS59 and DAVIS16.
arXiv Detail & Related papers (2020-08-18T11:40:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.