FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architectures

• URL: http://arxiv.org/abs/2310.20593v1
• Date: Tue, 31 Oct 2023 16:30:16 GMT
• Title: FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architectures
• Authors: Andrea Ciamarra, Federico Becattini, Lorenzo Seidenari, Alberto Del Bimbo
• Abstract summary: We propose a flow and depth forecasting model, trained to jointly forecast both modalities at once. We train the proposed model to also perform predictions for several timesteps in the future. We report benefits on the downstream task of segmentation forecasting, injecting our predictions in a flow-based mask-warping framework.
• Score: 31.879514593973195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Forecasting motion and spatial positions of objects is of fundamental importance, especially in safety-critical settings such as autonomous driving. In this work, we address the issue by forecasting two different modalities that carry complementary information, namely optical flow and depth. To this end we propose FLODCAST a flow and depth forecasting model that leverages a multitask recurrent architecture, trained to jointly forecast both modalities at once. We stress the importance of training using flows and depth maps together, demonstrating that both tasks improve when the model is informed of the other modality. We train the proposed model to also perform predictions for several timesteps in the future. This provides better supervision and leads to more precise predictions, retaining the capability of the model to yield outputs autoregressively for any future time horizon. We test our model on the challenging Cityscapes dataset, obtaining state of the art results for both flow and depth forecasting. Thanks to the high quality of the generated flows, we also report benefits on the downstream task of segmentation forecasting, injecting our predictions in a flow-based mask-warping framework.

Related papers

• Physics-guided Active Sample Reweighting for Urban Flow Prediction [75.24539704456791]
  Urban flow prediction is a nuanced-temporal modeling that estimates the throughput of transportation services like buses, taxis and ride-driven models. Some recent prediction solutions bring remedies with the notion of physics-guided machine learning (PGML) We develop a atized physics-guided network (PN), and propose a data-aware framework Physics-guided Active Sample Reweighting (P-GASR)
  arXiv  Detail & Related papers  (2024-07-18T15:44:23Z)
• AdaOcc: Adaptive Forward View Transformation and Flow Modeling for 3D Occupancy and Flow Prediction [56.72301849123049]
  We present our solution for the Vision-Centric 3D Occupancy and Flow Prediction track in the nuScenes Open-Occ dataset challenge at CVPR 2024. Our innovative approach involves a dual-stage framework that enhances 3D occupancy and flow predictions by incorporating adaptive forward view transformation and flow modeling. Our method combines regression with classification to address scale variations in different scenes, and leverages predicted flow to warp current voxel features to future frames, guided by future frame ground truth.
  arXiv  Detail & Related papers  (2024-07-01T16:32:15Z)
• Deep Vision-Based Framework for Coastal Flood Prediction Under Climate Change Impacts and Shoreline Adaptations [0.3413711585591077]
  We present a systematic framework for training high-fidelity Deep Vision-based coastal flood prediction models in low-data settings. We also introduce a deep CNN architecture tailored specifically to the coastal flood prediction problem at hand. The performance of the developed DL models is validated against commonly adopted geostatistical regression methods.
  arXiv  Detail & Related papers  (2024-06-06T19:54:34Z)
• DeTra: A Unified Model for Object Detection and Trajectory Forecasting [68.85128937305697]
  Our approach formulates the union of the two tasks as a trajectory refinement problem. To tackle this unified task, we design a refinement transformer that infers the presence, pose, and multi-modal future behaviors of objects. In our experiments, we observe that ourmodel outperforms the state-of-the-art on Argoverse 2 Sensor and Open dataset.
  arXiv  Detail & Related papers  (2024-06-06T18:12:04Z)
• A Multi-Channel Spatial-Temporal Transformer Model for Traffic Flow Forecasting [0.0]
  We propose a multi-channel spatial-temporal transformer model for traffic flow forecasting. It improves the accuracy of the prediction by fusing results from different channels of traffic data. Experimental results on six real-world datasets demonstrate that introducing a multi-channel mechanism into the temporal model enhances performance.
  arXiv  Detail & Related papers  (2024-05-10T06:37:07Z)
• Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving [68.95178518732965]
  A self-driving vehicle (SDV) must be able to perceive its surroundings and predict the future behavior of other traffic participants. Existing works either perform object detection followed by trajectory of the detected objects, or predict dense occupancy and flow grids for the whole scene. This motivates our unified approach to perception and future prediction that implicitly represents occupancy and flow over time with a single neural network.
  arXiv  Detail & Related papers  (2023-08-02T23:39:24Z)
• Forecasting Future Instance Segmentation with Learned Optical Flow and Warping [31.879514593973195]
  In this paper we investigate the usage of optical flow for predicting future semantic segmentations. Results on the Cityscapes dataset demonstrate the effectiveness of optical-flow methods.
  arXiv  Detail & Related papers  (2022-11-15T11:01:12Z)
• Learning Accurate Dense Correspondences and When to Trust Them [161.76275845530964]
  We aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. Our approach obtains state-of-the-art results on challenging geometric matching and optical flow datasets.
  arXiv  Detail & Related papers  (2021-01-05T18:54:11Z)
• Deep multi-stations weather forecasting: explainable recurrent convolutional neural networks [4.213427823201119]
  We show that adding a self-attention within the models increases the overall forecasting performance. The present paper compares two different deep learning architectures to perform weather prediction on daily gathered data from 18 cities across Europe.
  arXiv  Detail & Related papers  (2020-09-23T16:22:25Z)
• The Importance of Prior Knowledge in Precise Multimodal Prediction [71.74884391209955]
  Roads have well defined geometries, topologies, and traffic rules. In this paper we propose to incorporate structured priors as a loss function. We demonstrate the effectiveness of our approach on real-world self-driving datasets.
  arXiv  Detail & Related papers  (2020-06-04T03:56:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.