Fake & Square: Training Self-Supervised Vision Transformers with Synthetic Data and Synthetic Hard Negatives

• URL: http://arxiv.org/abs/2509.02029v1
• Date: Tue, 02 Sep 2025 07:17:46 GMT
• Title: Fake & Square: Training Self-Supervised Vision Transformers with Synthetic Data and Synthetic Hard Negatives
• Abstract summary: We build on existing self-supervised learning approaches for vision, drawing inspiration from the adage "fake it till you make it"<n>We study the potential of generative models for unsupervised representation learning, leveraging synthetic data to augment sample diversity.<n>Second, we examine the feasibility of generating synthetic hard negatives in the representation space, creating diverse and challenging contrasts.
• Score: 5.571793666361683
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper does not introduce a new method per se. Instead, we build on existing self-supervised learning approaches for vision, drawing inspiration from the adage "fake it till you make it". While contrastive self-supervised learning has achieved remarkable success, it typically relies on vast amounts of real-world data and carefully curated hard negatives. To explore alternatives to these requirements, we investigate two forms of "faking it" in vision transformers. First, we study the potential of generative models for unsupervised representation learning, leveraging synthetic data to augment sample diversity. Second, we examine the feasibility of generating synthetic hard negatives in the representation space, creating diverse and challenging contrasts. Our framework - dubbed Syn2Co - combines both approaches and evaluates whether synthetically enhanced training can lead to more robust and transferable visual representations on DeiT-S and Swin-T architectures. Our findings highlight the promise and limitations of synthetic data in self-supervised learning, offering insights for future work in this direction.

Related papers

• From Synthetic to Real: Toward Identity-Consistent Makeup Transfer with Synthetic and Real Data [63.40634698361175]
  Makeup transfer aims to apply the makeup style of a reference portrait to a source portrait while preserving identity and background.<n>Recent diffusion- and flow-based approaches instead exploit synthetic data for supervised training, leading to significant improvements.<n>This paper proposes ConsistentBeauty and RealBeauty, a synthetic-to-real post-training framework.
  arXiv  Detail & Related papers  (2026-05-08T15:21:43Z)
• When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators [8.125052877783043]
  This work revisits the promise of synthetic data as a scalable substitute for real training sets.<n>We generate large-scale synthetic datasets using state-of-the-art T2I models released between 2022 and 2025.<n>Despite observable advances in visual fidelity and prompt adherence, classification accuracy on real test data consistently declines with newer T2I models as training data generators.
  arXiv  Detail & Related papers  (2026-02-23T15:15:53Z)
• Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data [53.040873127309766]
  We propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning.<n>Our method outperforms existing models on both in-dataset and cross-dataset evaluations.
  arXiv  Detail & Related papers  (2025-09-08T17:58:06Z)
• Unsupervised Training of Vision Transformers with Synthetic Negatives [5.571793666361683]
  We integrate synthetic hard negatives to improve vision transformer representation learning.<n>This simple yet effective technique notably improves the discriminative power of learned representations.
  arXiv  Detail & Related papers  (2025-09-02T07:14:21Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training [0.850206009406913]
  MViTac is a novel methodology that leverages contrastive learning to integrate vision and touch sensations in a self-supervised fashion. By availing both sensory inputs, MViTac leverages intra and inter-modality losses for learning representations, resulting in enhanced material property classification and more adept grasping prediction.
  arXiv  Detail & Related papers  (2024-01-22T15:11:57Z)
• Learning Robust Representations via Bidirectional Transition for Visual Reinforcement Learning [49.23256535551141]
  We introduce a Bidirectional Transition (BiT) model, which leverages the ability to bidirectionally predict environmental transitions both forward and backward to extract reliable representations.<n>Our model demonstrates competitive generalization performance and sample efficiency on two settings of the DeepMind Control suite.
  arXiv  Detail & Related papers  (2023-12-04T14:19:36Z)
• ContraNeRF: Generalizable Neural Radiance Fields for Synthetic-to-real Novel View Synthesis via Contrastive Learning [102.46382882098847]
  We first investigate the effects of synthetic data in synthetic-to-real novel view synthesis. We propose to introduce geometry-aware contrastive learning to learn multi-view consistent features with geometric constraints. Our method can render images with higher quality and better fine-grained details, outperforming existing generalizable novel view synthesis methods in terms of PSNR, SSIM, and LPIPS.
  arXiv  Detail & Related papers  (2023-03-20T12:06:14Z)
• A Scaling Law for Synthetic-to-Real Transfer: A Measure of Pre-Training [52.93808218720784]
  Synthetic-to-real transfer learning is a framework in which we pre-train models with synthetically generated images and ground-truth annotations for real tasks. Although synthetic images overcome the data scarcity issue, it remains unclear how the fine-tuning performance scales with pre-trained models. We observe a simple and general scaling law that consistently describes learning curves in various tasks, models, and complexities of synthesized pre-training data.
  arXiv  Detail & Related papers  (2021-08-25T02:29:28Z)
• A Broad Study on the Transferability of Visual Representations with Contrastive Learning [15.667240680328922]
  We study the transferability of learned representations of contrastive approaches for linear evaluation, full-network transfer, and few-shot recognition. The results show that the contrastive approaches learn representations that are easily transferable to a different downstream task. Our analysis reveals that the representations learned from the contrastive approaches contain more low/mid-level semantics than cross-entropy models.
  arXiv  Detail & Related papers  (2021-03-24T22:55:04Z)
• Unsupervised Landmark Learning from Unpaired Data [117.81440795184587]
  Recent attempts for unsupervised landmark learning leverage synthesized image pairs that are similar in appearance but different in poses. We propose a cross-image cycle consistency framework which applies the swapping-reconstruction strategy twice to obtain the final supervision. Our proposed framework is shown to outperform strong baselines by a large margin.
  arXiv  Detail & Related papers  (2020-06-29T13:57:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.