SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving

• URL: http://arxiv.org/abs/2303.06209v2
• Date: Tue, 8 Aug 2023 08:06:48 GMT
• Title: SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving
• Authors: Shuai Yuan, Shuzhi Yu, Hannah Kim and Carlo Tomasi
• Abstract summary: We introduce SemARFlow, an unsupervised optical flow network designed for autonomous driving data. We show visible improvements around object boundaries as well as a greater ability to generalize across datasets.
• Score: 5.342413115295559
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised optical flow estimation is especially hard near occlusions and motion boundaries and in low-texture regions. We show that additional information such as semantics and domain knowledge can help better constrain this problem. We introduce SemARFlow, an unsupervised optical flow network designed for autonomous driving data that takes estimated semantic segmentation masks as additional inputs. This additional information is injected into the encoder and into a learned upsampler that refines the flow output. In addition, a simple yet effective semantic augmentation module provides self-supervision when learning flow and its boundaries for vehicles, poles, and sky. Together, these injections of semantic information improve the KITTI-2015 optical flow test error rate from 11.80% to 8.38%. We also show visible improvements around object boundaries as well as a greater ability to generalize across datasets. Code is available at https://github.com/duke-vision/semantic-unsup-flow-release.

Related papers

• UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model [12.706915226843401]
  UnSAMFlow is an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM) We analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. Our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on KITTI and Sintel datasets.
  arXiv  Detail & Related papers  (2024-05-04T08:27:12Z)
• FlowDA: Unsupervised Domain Adaptive Framework for Optical Flow Estimation [6.122542233250026]
  This paper introduces FlowDA, an unsupervised domain adaptive (UDA) framework for optical flow estimation. FlowDA outperforms state-of-the-art unsupervised optical flow estimation method SMURF by 21.6%, real optical flow dataset generation method MPI-Flow by 27.8%, and optical flow estimation adaptive method FlowSupervisor by 30.9%.
  arXiv  Detail & Related papers  (2023-12-28T12:51:48Z)
• GAFlow: Incorporating Gaussian Attention into Optical Flow [62.646389181507764]
  We push Gaussian Attention (GA) into the optical flow models to accentuate local properties during representation learning. We introduce a novel Gaussian-Constrained Layer (GCL) which can be easily plugged into existing Transformer blocks. For reliable motion analysis, we provide a new Gaussian-Guided Attention Module (GGAM)
  arXiv  Detail & Related papers  (2023-09-28T07:46:01Z)
• TransFlow: Transformer as Flow Learner [22.727953339383344]
  We propose TransFlow, a pure transformer architecture for optical flow estimation. It provides more accurate correlation and trustworthy matching in flow estimation. It recovers more compromised information in flow estimation through long-range temporal association in dynamic scenes.
  arXiv  Detail & Related papers  (2023-04-23T03:11:23Z)
• DistractFlow: Improving Optical Flow Estimation via Realistic Distractions and Pseudo-Labeling [49.46842536813477]
  We propose a novel data augmentation approach, DistractFlow, for training optical flow estimation models. We combine one of the frames in the pair with a distractor image depicting a similar domain, which allows for inducing visual perturbations congruent with natural objects and scenes. Our approach allows increasing the number of available training pairs significantly without requiring additional annotations.
  arXiv  Detail & Related papers  (2023-03-24T15:42:54Z)
• Semi-Supervised Learning of Optical Flow by Flow Supervisor [16.406213579356795]
  We propose a practical fine tuning method to adapt a pretrained model to a target dataset without ground truth flows. This design is aimed at stable convergence and better accuracy over conventional self-supervision methods. We achieve meaningful improvements over state-of-the-art optical flow models on Sintel and KITTI benchmarks.
  arXiv  Detail & Related papers  (2022-07-21T06:11:52Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• CSFlow: Learning Optical Flow via Cross Strip Correlation for Autonomous Driving [9.562270891742982]
  Cross Strip Correlation module (CSC) and Correlation Regression Initialization module (CRI) CSFlow consists of two novel modules: Cross Strip Correlation module (CSC) and Correlation Regression Initialization module (CRI)
  arXiv  Detail & Related papers  (2022-02-02T08:17:45Z)
• Sensor-Guided Optical Flow [53.295332513139925]
  This paper proposes a framework to guide an optical flow network with external cues to achieve superior accuracy on known or unseen domains. We show how these can be obtained by combining depth measurements from active sensors with geometry and hand-crafted optical flow algorithms.
  arXiv  Detail & Related papers  (2021-09-30T17:59:57Z)
• AutoFlow: Learning a Better Training Set for Optical Flow [62.40293188964933]
  AutoFlow is a method to render training data for optical flow. AutoFlow achieves state-of-the-art accuracy in pre-training both PWC-Net and RAFT.
  arXiv  Detail & Related papers  (2021-04-29T17:55:23Z)
• What Matters in Unsupervised Optical Flow [51.45112526506455]
  We compare and analyze a set of key components in unsupervised optical flow. We construct a number of novel improvements to unsupervised flow models. We present a new unsupervised flow technique that significantly outperforms the previous state-of-the-art.
  arXiv  Detail & Related papers  (2020-06-08T19:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.