StreamingFlow: Streaming Occupancy Forecasting with Asynchronous Multi-modal Data Streams via Neural Ordinary Differential Equation

• URL: http://arxiv.org/abs/2302.09585v2
• Date: Tue, 11 Jun 2024 07:42:12 GMT
• Title: StreamingFlow: Streaming Occupancy Forecasting with Asynchronous Multi-modal Data Streams via Neural Ordinary Differential Equation
• Authors: Yining Shi, Kun Jiang, Ke Wang, Jiusi Li, Yunlong Wang, Mengmeng Yang, Diange Yang,
• Abstract summary: StreamingFlow is a novel BEV occupancy predictor that ingests asynchronous multi-sensor data streams for fusion. It learns derivatives of BEV features over temporal horizons, updates the implicit sensor's BEV features as part of the fusion process, and propagates BEV states to the desired future time point. It significantly outperforms previous vision-based, LiDAR-based methods, and shows superior performance compared to state-of-the-art fusion-based methods.
• Score: 15.441175735210791
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting the future occupancy states of the surrounding environment is a vital task for autonomous driving. However, current best-performing single-modality methods or multi-modality fusion perception methods are only able to predict uniform snapshots of future occupancy states and require strictly synchronized sensory data for sensor fusion. We propose a novel framework, StreamingFlow, to lift these strong limitations. StreamingFlow is a novel BEV occupancy predictor that ingests asynchronous multi-sensor data streams for fusion and performs streaming forecasting of the future occupancy map at any future timestamps. By integrating neural ordinary differential equations (N-ODE) into recurrent neural networks, StreamingFlow learns derivatives of BEV features over temporal horizons, updates the implicit sensor's BEV features as part of the fusion process, and propagates BEV states to the desired future time point. It shows good zero-shot generalization ability of prediction, reflected in the interpolation of the observed prediction time horizon and the reasonable inference of the unseen farther future period. Extensive experiments on two large-scale datasets, nuScenes and Lyft L5, demonstrate that StreamingFlow significantly outperforms previous vision-based, LiDAR-based methods, and shows superior performance compared to state-of-the-art fusion-based methods.

Related papers

• SuperFlow++: Enhanced Spatiotemporal Consistency for Cross-Modal Data Pretraining [62.433137130087445]
  SuperFlow++ is a novel framework that integrates pretraining and downstream tasks using consecutive camera pairs. We show that SuperFlow++ outperforms state-of-the-art methods across diverse tasks and driving conditions. With strong generalizability and computational efficiency, SuperFlow++ establishes a new benchmark for data-efficient LiDAR-based perception in autonomous driving.
  arXiv  Detail & Related papers  (2025-03-25T17:59:57Z)
• Probabilistic Forecasting via Autoregressive Flow Matching [1.5467259918426441]
  FlowTime is a generative model for probabilistic forecasting of timeseries data. We decompose the joint distribution of future observations into a sequence of conditional densities, each modeled via a shared flow. We demonstrate the effectiveness of FlowTime on multiple dynamical systems and real-world forecasting tasks.
  arXiv  Detail & Related papers  (2025-03-13T13:54:24Z)
• ODEStream: A Buffer-Free Online Learning Framework with ODE-based Adaptor for Streaming Time Series Forecasting [11.261457967759688]
  ODEStream is a buffer-free continual learning framework that incorporates a temporal isolation layer to capture temporal dependencies within the data. It generates a continuous data representation, enabling seamless adaptation to changing dynamics in a data streaming scenario. Our approach focuses on learning how the dynamics and distribution of historical data change over time, facilitating direct processing of streaming sequences.
  arXiv  Detail & Related papers  (2024-11-11T22:36:33Z)
• Physics-guided Active Sample Reweighting for Urban Flow Prediction [75.24539704456791]
  Urban flow prediction is a nuanced-temporal modeling that estimates the throughput of transportation services like buses, taxis and ride-driven models. Some recent prediction solutions bring remedies with the notion of physics-guided machine learning (PGML) We develop a atized physics-guided network (PN), and propose a data-aware framework Physics-guided Active Sample Reweighting (P-GASR)
  arXiv  Detail & Related papers  (2024-07-18T15:44:23Z)
• BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents [56.33989853438012]
  We propose BEVWorld, a framework that transforms multimodal sensor inputs into a unified and compact Bird's Eye View latent space for holistic environment modeling. The proposed world model consists of two main components: a multi-modal tokenizer and a latent BEV sequence diffusion model.
  arXiv  Detail & Related papers  (2024-07-08T07:26:08Z)
• MemFlow: Optical Flow Estimation and Prediction with Memory [54.22820729477756]
  We present MemFlow, a real-time method for optical flow estimation and prediction with memory. Our method enables memory read-out and update modules for aggregating historical motion information in real-time. Our approach seamlessly extends to the future prediction of optical flow based on past observations.
  arXiv  Detail & Related papers  (2024-04-07T04:56:58Z)
• Adapting to Length Shift: FlexiLength Network for Trajectory Prediction [53.637837706712794]
  Trajectory prediction plays an important role in various applications, including autonomous driving, robotics, and scene understanding. Existing approaches mainly focus on developing compact neural networks to increase prediction precision on public datasets, typically employing a standardized input duration. We introduce a general and effective framework, the FlexiLength Network (FLN), to enhance the robustness of existing trajectory prediction against varying observation periods.
  arXiv  Detail & Related papers  (2024-03-31T17:18:57Z)
• GDTS: Goal-Guided Diffusion Model with Tree Sampling for Multi-Modal Pedestrian Trajectory Prediction [15.731398013255179]
  We propose a novel Goal-Guided Diffusion Model with Tree Sampling for multi-modal trajectory prediction. A two-stage tree sampling algorithm is presented, which leverages common features to reduce the inference time and improve accuracy for multi-modal prediction. Experimental results demonstrate that our proposed framework achieves comparable state-of-the-art performance with real-time inference speed in public datasets.
  arXiv  Detail & Related papers  (2023-11-25T03:55:06Z)
• Streaming Factor Trajectory Learning for Temporal Tensor Decomposition [33.18423605559094]
  We propose Streaming Factor Trajectory Learning for temporal tensor decomposition. We use Gaussian processes (GPs) to model the trajectory of factors so as to flexibly estimate their temporal evolution. We have shown the advantage of SFTL in both synthetic tasks and real-world applications.
  arXiv  Detail & Related papers  (2023-10-25T21:58:52Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• A Novel Deep Neural Network for Trajectory Prediction in Automated Vehicles Using Velocity Vector Field [12.067838086415833]
  This paper proposes a novel technique for trajectory prediction that combines a data-driven learning-based method with a velocity vector field (VVF) generated from a nature-inspired concept. The accuracy remains consistent with decreasing observation windows which alleviates the requirement of a long history of past observations for accurate trajectory prediction.
  arXiv  Detail & Related papers  (2023-09-19T22:14:52Z)
• Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement [51.55157852647306]
  Time series forecasting has been a widely explored task of great importance in many applications. It is common that real-world time series data are recorded in a short time period, which results in a big gap between the deep model and the limited and noisy time series. We propose to address the time series forecasting problem with generative modeling and propose a bidirectional variational auto-encoder equipped with diffusion, denoise, and disentanglement.
  arXiv  Detail & Related papers  (2023-01-08T12:20:46Z)
• Real-time Object Detection for Streaming Perception [84.2559631820007]
  Streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. We build a simple and effective framework for streaming perception. Our method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline.
  arXiv  Detail & Related papers  (2022-03-23T11:33:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.