Transferring disentangled representations: bridging the gap between synthetic and real images

• URL: http://arxiv.org/abs/2409.18017v1
• Date: Thu, 26 Sep 2024 16:25:48 GMT
• Title: Transferring disentangled representations: bridging the gap between synthetic and real images
• Authors: Jacopo Dapueto, Nicoletta Noceti, Francesca Odone,
• Abstract summary: We investigate the possibility of leveraging synthetic data to learn general-purpose disentangled representations applicable to real data. We propose a new interpretable intervention-based metric, to measure the quality of factors encoding in the representation. Our results indicate that some level of disentanglement, transferring a representation from synthetic to real data, is possible and effective.
• Score: 1.0760018917783072
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developing meaningful and efficient representations that separate the fundamental structure of the data generation mechanism is crucial in representation learning. However, Disentangled Representation Learning has not fully shown its potential on real images, because of correlated generative factors, their resolution and limited access to ground truth labels. Specifically on the latter, we investigate the possibility of leveraging synthetic data to learn general-purpose disentangled representations applicable to real data, discussing the effect of fine-tuning and what properties of disentanglement are preserved after the transfer. We provide an extensive empirical study to address these issues. In addition, we propose a new interpretable intervention-based metric, to measure the quality of factors encoding in the representation. Our results indicate that some level of disentanglement, transferring a representation from synthetic to real data, is possible and effective.

Related papers

• Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data [2.6016285265085526]
  Student models show a significant drop in accuracy compared to models trained on real data. By training these layers using either real or synthetic data, we reveal that the drop mainly stems from the model's final layers. Our results suggest an improved trade-off between the amount of real training data used and the model's accuracy.
  arXiv  Detail & Related papers  (2024-05-06T07:51:13Z)
• Massively Annotated Datasets for Assessment of Synthetic and Real Data in Face Recognition [0.2775636978045794]
  We study the drift between the performance of models trained on real and synthetic datasets. We conduct studies on the differences between real and synthetic datasets on the attribute set. Interestingly enough, we have verified that while real samples suffice to explain the synthetic distribution, the opposite could not be further from being true.
  arXiv  Detail & Related papers  (2024-04-23T17:10:49Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
  We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
  arXiv  Detail & Related papers  (2023-11-03T03:18:40Z)
• Flow Factorized Representation Learning [109.51947536586677]
  We introduce a generative model which specifies a distinct set of latent probability paths that define different input transformations. We show that our model achieves higher likelihoods on standard representation learning benchmarks while simultaneously being closer to approximately equivariant models.
  arXiv  Detail & Related papers  (2023-09-22T20:15:37Z)
• Knowing the Distance: Understanding the Gap Between Synthetic and Real Data For Face Parsing [0.0]
  We show that the distribution gap is the largest contributor to the performance gap, accounting for over 50% of the gap. This suggests that synthetic data is a viable alternative to real data, especially when real data is limited or difficult to obtain.
  arXiv  Detail & Related papers  (2023-03-27T13:59:26Z)
• Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performances [76.34037366117234]
  We introduce a new dataset called Robot Control Gestures (RoCoG-v2) The dataset is composed of both real and synthetic videos from seven gesture classes. We present results using state-of-the-art action recognition and domain adaptation algorithms.
  arXiv  Detail & Related papers  (2023-03-17T23:23:55Z)
• A Scaling Law for Synthetic-to-Real Transfer: A Measure of Pre-Training [52.93808218720784]
  Synthetic-to-real transfer learning is a framework in which we pre-train models with synthetically generated images and ground-truth annotations for real tasks. Although synthetic images overcome the data scarcity issue, it remains unclear how the fine-tuning performance scales with pre-trained models. We observe a simple and general scaling law that consistently describes learning curves in various tasks, models, and complexities of synthesized pre-training data.
  arXiv  Detail & Related papers  (2021-08-25T02:29:28Z)
• Attention-based Adversarial Appearance Learning of Augmented Pedestrians [49.25430012369125]
  We propose a method to synthesize realistic data for the pedestrian recognition task. Our approach utilizes an attention mechanism driven by an adversarial loss to learn domain discrepancies. Our experiments confirm that the proposed adaptation method is robust to such discrepancies and reveals both visual realism and semantic consistency.
  arXiv  Detail & Related papers  (2021-07-06T15:27:00Z)
• On the Transfer of Disentangled Representations in Realistic Settings [44.367245337475445]
  We introduce a new high-resolution dataset with 1M simulated images and over 1,800 annotated real-world images. We propose new architectures in order to scale disentangled representation learning to realistic high-resolution settings.
  arXiv  Detail & Related papers  (2020-10-27T16:15:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.