The Devil in the Details: Simple and Effective Optical Flow Synthetic Data Generation

• URL: http://arxiv.org/abs/2308.07378v1
• Date: Mon, 14 Aug 2023 18:01:45 GMT
• Title: The Devil in the Details: Simple and Effective Optical Flow Synthetic Data Generation
• Authors: Kwon Byung-Ki, Kim Sung-Bin, Tae-Hyun Oh
• Abstract summary: We show that the required characteristics in an optical flow dataset are rather simple and present a simpler synthetic data generation method. With 2D motion-based datasets, we systematically analyze the simplest yet critical factors for generating synthetic datasets.
• Score: 19.945859289278534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work on dense optical flow has shown significant progress, primarily in a supervised learning manner requiring a large amount of labeled data. Due to the expensiveness of obtaining large scale real-world data, computer graphics are typically leveraged for constructing datasets. However, there is a common belief that synthetic-to-real domain gaps limit generalization to real scenes. In this paper, we show that the required characteristics in an optical flow dataset are rather simple and present a simpler synthetic data generation method that achieves a certain level of realism with compositions of elementary operations. With 2D motion-based datasets, we systematically analyze the simplest yet critical factors for generating synthetic datasets. Furthermore, we propose a novel method of utilizing occlusion masks in a supervised method and observe that suppressing gradients on occluded regions serves as a powerful initial state in the curriculum learning sense. The RAFT network initially trained on our dataset outperforms the original RAFT on the two most challenging online benchmarks, MPI Sintel and KITTI 2015.

Related papers

• A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation [42.2398858786125]
  Deep learning in computer vision has achieved great success with the price of large-scale labeled training data. The uncontrollable data collection process produces non-IID training and test data, where undesired duplication may exist. To circumvent them, an alternative is to generate synthetic data via 3D rendering with domain randomization.
  arXiv  Detail & Related papers  (2023-03-16T09:03:52Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets [83.749895930242]
  We propose two techniques for producing high-quality naturalistic synthetic occluded faces. We empirically show the effectiveness and robustness of both methods, even for unseen occlusions. We present two high-resolution real-world occluded face datasets with fine-grained annotations, RealOcc and RealOcc-Wild.
  arXiv  Detail & Related papers  (2022-05-12T17:03:57Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)
• A Scaling Law for Synthetic-to-Real Transfer: A Measure of Pre-Training [52.93808218720784]
  Synthetic-to-real transfer learning is a framework in which we pre-train models with synthetically generated images and ground-truth annotations for real tasks. Although synthetic images overcome the data scarcity issue, it remains unclear how the fine-tuning performance scales with pre-trained models. We observe a simple and general scaling law that consistently describes learning curves in various tasks, models, and complexities of synthesized pre-training data.
  arXiv  Detail & Related papers  (2021-08-25T02:29:28Z)
• Optical Flow Dataset Synthesis from Unpaired Images [36.158607790844705]
  We introduce a novel method to build a training set of pseudo-real images that can be used to train optical flow in a supervised manner. Our dataset uses two unpaired frames from real data and creates pairs of frames by simulating random warps. We thus obtain the benefit of directly training on real data while having access to an exact ground truth.
  arXiv  Detail & Related papers  (2021-04-02T22:19:47Z)
• Semi-synthesis: A fast way to produce effective datasets for stereo matching [16.602343511350252]
  Close-to-real-scene texture rendering is a key factor to boost up stereo matching performance. We propose semi-synthetic, an effective and fast way to synthesize large amount of data with close-to-real-scene texture. With further fine-tuning on the real dataset, we also achieve SOTA performance on Middlebury and competitive results on KITTI and ETH3D datasets.
  arXiv  Detail & Related papers  (2021-01-26T14:34:49Z)
• On the Transfer of Disentangled Representations in Realistic Settings [44.367245337475445]
  We introduce a new high-resolution dataset with 1M simulated images and over 1,800 annotated real-world images. We propose new architectures in order to scale disentangled representation learning to realistic high-resolution settings.
  arXiv  Detail & Related papers  (2020-10-27T16:15:24Z)
• Deep SVBRDF Estimation on Real Materials [5.37133760455631]
  We show that training networks exclusively on synthetic data is insufficient to achieve adequate results when tested on real data. We show that adapting network weights to real data is of critical importance, resulting in an approach which significantly outperforms previous methods for SVBRDF estimation on real materials.
  arXiv  Detail & Related papers  (2020-10-08T17:41:26Z)
• Syn2Real Transfer Learning for Image Deraining using Gaussian Processes [92.15895515035795]
  CNN-based methods for image deraining have achieved excellent performance in terms of reconstruction error as well as visual quality. Due to challenges in obtaining real world fully-labeled image deraining datasets, existing methods are trained only on synthetically generated data. We propose a Gaussian Process-based semi-supervised learning framework which enables the network in learning to derain using synthetic dataset.
  arXiv  Detail & Related papers  (2020-06-10T00:33:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.