UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model

• URL: http://arxiv.org/abs/2405.02608v1
• Date: Sat, 4 May 2024 08:27:12 GMT
• Title: UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model
• Authors: Shuai Yuan, Lei Luo, Zhuo Hui, Can Pu, Xiaoyu Xiang, Rakesh Ranjan, Denis Demandolx,
• Abstract summary: UnSAMFlow is an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM) We analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. Our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on KITTI and Sintel datasets.
• Score: 12.706915226843401
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Traditional unsupervised optical flow methods are vulnerable to occlusions and motion boundaries due to lack of object-level information. Therefore, we propose UnSAMFlow, an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM). We first include a self-supervised semantic augmentation module tailored to SAM masks. We also analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. A simple yet effective mask feature module has also been added to further aggregate features on the object level. With all these adaptations, our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on both KITTI and Sintel datasets. Our method also generalizes well across domains and runs very efficiently.

Related papers

• MaskFlow: Object-Aware Motion Estimation [0.45646200630189254]
  We introduce a novel motion estimation method, MaskFlow, that is capable of estimating accurate motion fields. In addition to lower-level features, that are used in other Deep Neural Network (DNN)-based motion estimation methods, MaskFlow draws from object-level features and segmentations.
  arXiv  Detail & Related papers  (2023-11-21T09:37:49Z)
• GAFlow: Incorporating Gaussian Attention into Optical Flow [62.646389181507764]
  We push Gaussian Attention (GA) into the optical flow models to accentuate local properties during representation learning. We introduce a novel Gaussian-Constrained Layer (GCL) which can be easily plugged into existing Transformer blocks. For reliable motion analysis, we provide a new Gaussian-Guided Attention Module (GGAM)
  arXiv  Detail & Related papers  (2023-09-28T07:46:01Z)
• SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model [17.88914104216893]
  We propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%.
  arXiv  Detail & Related papers  (2023-07-31T11:40:53Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving [5.342413115295559]
  We introduce SemARFlow, an unsupervised optical flow network designed for autonomous driving data. We show visible improvements around object boundaries as well as a greater ability to generalize across datasets.
  arXiv  Detail & Related papers  (2023-03-10T21:17:14Z)
• Exploiting Shape Cues for Weakly Supervised Semantic Segmentation [15.791415215216029]
  Weakly supervised semantic segmentation (WSSS) aims to produce pixel-wise class predictions with only image-level labels for training. We propose to exploit shape information to supplement the texture-biased property of convolutional neural networks (CNNs) We further refine the predictions in an online fashion with a novel refinement method that takes into account both the class and the color affinities.
  arXiv  Detail & Related papers  (2022-08-08T17:25:31Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency [114.02182755620784]
  We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular camera setup without supervision. Our framework is shown to outperform the state-of-the-art depth and motion estimation methods.
  arXiv  Detail & Related papers  (2021-02-04T14:26:42Z)
• What Matters in Unsupervised Optical Flow [51.45112526506455]
  We compare and analyze a set of key components in unsupervised optical flow. We construct a number of novel improvements to unsupervised flow models. We present a new unsupervised flow technique that significantly outperforms the previous state-of-the-art.
  arXiv  Detail & Related papers  (2020-06-08T19:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.