StereoGAN: Bridging Synthetic-to-Real Domain Gap by Joint Optimization of Domain Translation and Stereo Matching

• URL: http://arxiv.org/abs/2005.01927v1
• Date: Tue, 5 May 2020 03:11:38 GMT
• Title: StereoGAN: Bridging Synthetic-to-Real Domain Gap by Joint Optimization of Domain Translation and Stereo Matching
• Authors: Rui Liu, Chengxi Yang, Wenxiu Sun, Xiaogang Wang, Hongsheng Li
• Abstract summary: Large-scale synthetic datasets are beneficial to stereo matching but usually introduce known domain bias. We propose an end-to-end training framework with domain translation and stereo matching networks to tackle this challenge.
• Score: 56.95846963856928
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale synthetic datasets are beneficial to stereo matching but usually introduce known domain bias. Although unsupervised image-to-image translation networks represented by CycleGAN show great potential in dealing with domain gap, it is non-trivial to generalize this method to stereo matching due to the problem of pixel distortion and stereo mismatch after translation. In this paper, we propose an end-to-end training framework with domain translation and stereo matching networks to tackle this challenge. First, joint optimization between domain translation and stereo matching networks in our end-to-end framework makes the former facilitate the latter one to the maximum extent. Second, this framework introduces two novel losses, i.e., bidirectional multi-scale feature re-projection loss and correlation consistency loss, to help translate all synthetic stereo images into realistic ones as well as maintain epipolar constraints. The effective combination of above two contributions leads to impressive stereo-consistent translation and disparity estimation accuracy. In addition, a mode seeking regularization term is added to endow the synthetic-to-real translation results with higher fine-grained diversity. Extensive experiments demonstrate the effectiveness of the proposed framework on bridging the synthetic-to-real domain gap on stereo matching.

Related papers

• SyntStereo2Real: Edge-Aware GAN for Remote Sensing Image-to-Image Translation while Maintaining Stereo Constraint [1.8749305679160366]
  Current methods involve combining two networks, an unpaired image-to-image translation network and a stereo-matching network. We propose an edge-aware GAN-based network that effectively tackles both tasks simultaneously. We demonstrate that our model produces qualitatively and quantitatively superior results than existing models, and its applicability extends to diverse domains.
  arXiv  Detail & Related papers  (2024-04-14T14:58:52Z)
• Toward Real World Stereo Image Super-Resolution via Hybrid Degradation Model and Discriminator for Implied Stereo Image Information [10.957275128743529]
  Real-world stereo image super-resolution has a significant influence on enhancing the performance of computer vision systems. Existing methods for single-image super-resolution can be applied to improve stereo images. This paper proposes a novel approach that integrates a implicit stereo information discriminator and a hybrid degradation model.
  arXiv  Detail & Related papers  (2023-12-13T07:24:50Z)
• Revisiting Domain Generalized Stereo Matching Networks from a Feature Consistency Perspective [65.37571681370096]
  We propose a simple pixel-wise contrastive learning across the viewpoints. A stereo selective whitening loss is introduced to better preserve the stereo feature consistency across domains. Our method achieves superior performance over several state-of-the-art networks.
  arXiv  Detail & Related papers  (2022-03-21T11:21:41Z)
• AdaStereo: An Efficient Domain-Adaptive Stereo Matching Approach [50.855679274530615]
  We present a novel domain-adaptive approach called AdaStereo to align multi-level representations for deep stereo matching networks. Our models achieve state-of-the-art cross-domain performance on multiple benchmarks, including KITTI, Middlebury, ETH3D and DrivingStereo. Our method is robust to various domain adaptation settings, and can be easily integrated into quick adaptation application scenarios and real-world deployments.
  arXiv  Detail & Related papers  (2021-12-09T15:10:47Z)
• SMD-Nets: Stereo Mixture Density Networks [68.56947049719936]
  We propose Stereo Mixture Density Networks (SMD-Nets), a simple yet effective learning framework compatible with a wide class of 2D and 3D architectures. Specifically, we exploit bimodal mixture densities as output representation and show that this allows for sharp and precise disparity estimates near discontinuities. We carry out comprehensive experiments on a new high-resolution and highly realistic synthetic stereo dataset, consisting of stereo pairs at 8Mpx resolution, as well as on real-world stereo datasets.
  arXiv  Detail & Related papers  (2021-04-08T16:15:46Z)
• Geometry-Aware Unsupervised Domain Adaptation for Stereo Matching [0.7233897166339268]
  We propose an attention mechanism that aggregates features in the left and right views, called Stereoscopic Cross Attention (SCA) SCA makes it possible to preserve the geometric structure of a stereo image pair in the process of the image-to-image translation. We empirically demonstrate the effectiveness of the proposed unsupervised domain adaptation based on the image-to-image translation with SCA.
  arXiv  Detail & Related papers  (2021-03-26T08:53:36Z)
• AdaStereo: A Simple and Efficient Approach for Adaptive Stereo Matching [50.06646151004375]
  A novel domain-adaptive pipeline called AdaStereo aims to align multi-level representations for deep stereo matching networks. Our AdaStereo models achieve state-of-the-art cross-domain performance on multiple stereo benchmarks, including KITTI, Middlebury, ETH3D, and DrivingStereo.
  arXiv  Detail & Related papers  (2020-04-09T16:15:13Z)
• Asymmetric GANs for Image-to-Image Translation [62.49892218126542]
  Existing models for Generative Adversarial Networks (GANs) learn the mapping from the source domain to the target domain using a cycle-consistency loss. We propose an AsymmetricGAN model with both translation and reconstruction generators of unequal sizes and different parameter-sharing strategy. Experiments on both supervised and unsupervised generative tasks with 8 datasets show that AsymmetricGAN achieves superior model capacity and better generation performance.
  arXiv  Detail & Related papers  (2019-12-14T21:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.