Unsupervised Training of Vision Transformers with Synthetic Negatives

• URL: http://arxiv.org/abs/2509.02024v1
• Date: Tue, 02 Sep 2025 07:14:21 GMT
• Title: Unsupervised Training of Vision Transformers with Synthetic Negatives
• Abstract summary: We integrate synthetic hard negatives to improve vision transformer representation learning.<n>This simple yet effective technique notably improves the discriminative power of learned representations.
• Score: 5.571793666361683
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper does not introduce a novel method per se. Instead, we address the neglected potential of hard negative samples in self-supervised learning. Previous works explored synthetic hard negatives but rarely in the context of vision transformers. We build on this observation and integrate synthetic hard negatives to improve vision transformer representation learning. This simple yet effective technique notably improves the discriminative power of learned representations. Our experiments show performance improvements for both DeiT-S and Swin-T architectures.

Related papers

• ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives [5.571793666361683]
  We introduce ViTAMINS, a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality.<n>Our proposed negatives give rise to emergent properties, where learned representations contain explicit information about the semantic content of an image.<n>Our findings motivate reconsidering contrastive learning as a simpler yet powerful alternative to dominant generative and self-distillation approaches.
  arXiv  Detail & Related papers  (2026-09-01T10:39:25Z)
• Fake & Square: Training Self-Supervised Vision Transformers with Synthetic Data and Synthetic Hard Negatives [5.571793666361683]
  We build on existing self-supervised learning approaches for vision, drawing inspiration from the adage "fake it till you make it"<n>We study the potential of generative models for unsupervised representation learning, leveraging synthetic data to augment sample diversity.<n>Second, we examine the feasibility of generating synthetic hard negatives in the representation space, creating diverse and challenging contrasts.
  arXiv  Detail & Related papers  (2025-09-02T07:17:46Z)
• When recalling in-context, Transformers are not SSMs [17.2249234816671]
  We show that the choice of learning rate plays a critical role in the performance of modern recurrent models.<n>We then inspect 1-layer transformers, revealing that despite their poor performance, their training dynamics surprisingly resemble the formation of induction heads.
  arXiv  Detail & Related papers  (2025-08-26T13:45:08Z)
• Mechanistic Understandings of Representation Vulnerabilities and Engineering Robust Vision Transformers [1.1187085721899017]
  We study the sources of known representation vulnerabilities of vision transformers (ViT), where perceptually identical images can have very different representations.<n>We develop NeuroShield-ViT, a novel defense mechanism that strategically neutralizes vulnerable neurons in earlier layers to prevent the cascade of adversarial effects.<n>Our results shed new light on how adversarial effects propagate through ViT layers, while providing a promising approach to enhance the robustness of vision transformers against adversarial attacks.
  arXiv  Detail & Related papers  (2025-02-07T05:58:16Z)
• Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization [88.5582111768376]
  We study the optimization of a Transformer composed of a self-attention layer with softmax followed by a fully connected layer under gradient descent on a certain data distribution model. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noise ratio in the data model.
  arXiv  Detail & Related papers  (2024-09-28T13:24:11Z)
• BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning [13.056764072568749]
  Humans have a remarkable ability to acquire, assimilate, and transfer knowledge across tasks throughout their lifetime without catastrophic forgetting. We propose BiRT, a novel representation rehearsal-based continual learning approach using vision transformers.
  arXiv  Detail & Related papers  (2023-05-08T15:19:39Z)
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision [117.15012005163322]
  AugSelf is an auxiliary self-supervised loss that learns the difference of augmentation parameters between two randomly augmented samples. Our intuition is that AugSelf encourages to preserve augmentation-aware information in learned representations, which could be beneficial for their transferability. AugSelf can easily be incorporated into recent state-of-the-art representation learning methods with a negligible additional training cost.
  arXiv  Detail & Related papers  (2021-11-18T10:43:50Z)
• Efficient Vision Transformers via Fine-Grained Manifold Distillation [96.50513363752836]
  Vision transformer architectures have shown extraordinary performance on many computer vision tasks. Although the network performance is boosted, transformers are often required more computational resources. We propose to excavate useful information from the teacher transformer through the relationship between images and the divided patches.
  arXiv  Detail & Related papers  (2021-07-03T08:28:34Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)
• An Empirical Study of Training Self-Supervised Visual Transformers [70.27107708555185]
  We study the effects of several fundamental components for training self-supervised Visual Transformers. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable.
  arXiv  Detail & Related papers  (2021-04-05T17:59:40Z)
• On the Adversarial Robustness of Visual Transformers [129.29523847765952]
  This work provides the first and comprehensive study on the robustness of vision transformers (ViTs) against adversarial perturbations. Tested on various white-box and transfer attack settings, we find that ViTs possess better adversarial robustness when compared with convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2021-03-29T14:48:24Z)
• A Broad Study on the Transferability of Visual Representations with Contrastive Learning [15.667240680328922]
  We study the transferability of learned representations of contrastive approaches for linear evaluation, full-network transfer, and few-shot recognition. The results show that the contrastive approaches learn representations that are easily transferable to a different downstream task. Our analysis reveals that the representations learned from the contrastive approaches contain more low/mid-level semantics than cross-entropy models.
  arXiv  Detail & Related papers  (2021-03-24T22:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.