Learning from Semantic Dictionaries: Discriminative Codebook Contrastive Learning for Unified Visual Representation and Generation

• URL: http://arxiv.org/abs/2605.25012v1
• Date: Sun, 24 May 2026 11:32:30 GMT
• Title: Learning from Semantic Dictionaries: Discriminative Codebook Contrastive Learning for Unified Visual Representation and Generation
• Abstract summary: Discriminative and generative vision models excel in their respective domains but remain semantically misaligned.<n>We introduce LEASE, a self-supervised framework that bridges this gap using a paired generative-discriminative codebook design.<n>On ImageNet-1K, LEASE achieves state-of-the-art unified performance, outperforming prior VQGAN-based methods.
• Score: 13.939029266977235
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Discriminative and generative vision models excel in their respective domains but remain semantically misaligned, hindering progress toward unified visual learning. We introduce LEASE (LEArning from SEmantic Dictionaries), a self-supervised framework that bridges this gap using a paired generative-discriminative codebook design. LEASE operates entirely in a discrete token space produced through a one-time precomputation step, enabling efficient training without data augmentations, teacher models, or online tokenizers. LEASE integrates two complementary objectives: a masked token reconstruction loss that captures fine-grained generative detail, and a codebook contrast loss that aligns encoder features with discriminative semantics via adaptive centroid weighting. This dual supervision yields a unified latent space that supports both high-quality generation and strong representation learning. On ImageNet-1K, LEASE achieves state-of-the-art unified performance, outperforming prior VQGAN-based methods such as MAGE and Sorcen across linear probing (up to +1.7%), unconditional generation (-1.26 FID and +10.19 IS w.r.t MAGE), few-shot learning (+0.56% on average against Sorcen), transfer (+0.75% average improvement against MAGE and Sorcen), and robustness benchmarks (+5.86% and +4.25% average improvement against MAGE and Sorcen, respectively). It also competes favorably with domain-specialized contrastive and generative models while surpassing previous MIM methods. The unsupervised LEASE model can also be extended to conditional generation by building upon its learned representations, proving competitive with specialized baselines. Overall, LEASE provides an efficient and effective step toward general-purpose vision models that jointly understand and generate visual content.

Related papers

• DREAM: Where Visual Understanding Meets Text-to-Image Generation [28.847476510280757]
  We introduce DREAM, a unified framework that jointly optimize discriminative and generative objectives.<n>We show that DREAM achieves 72.7% ImageNet linear-probing accuracy (+1.1% over CLIP) and an FID of 4.25 (+6.2% over FLUID)<n>Results demonstrate that discriminative and generative objectives can be synergistic, allowing unified multimodal models that excel at both visual understanding and generation.
  arXiv  Detail & Related papers  (2026-03-03T06:54:19Z)
• On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding [53.55135022958052]
  We propose Generation-Assisted Discriminative Discriminative(GAD) for closed-set action understanding.<n>GAD operates only during fine-tuning, preserving full compatibility with MLLM pretraining.<n>Experiments on temporal action understanding benchmarks demonstrate that GAD improves both accuracy and efficiency over generative methods.
  arXiv  Detail & Related papers  (2026-03-03T03:02:01Z)
• GRACE: Generative Representation Learning via Contrastive Policy Optimization [44.40343928975602]
  We introduce GRACE, a framework that reimagines contrastive signals as rewards for a generative policy.<n>GRACE produces explicit, human-interpretable rationales--structured natural language explanations.<n>On MTEB benchmark, GRACE yields broad cross category gains.
  arXiv  Detail & Related papers  (2025-10-06T05:46:56Z)
• Variational Supervised Contrastive Learning [50.79938854370321]
  We propose Variational Supervised Contrastive Learning (VarCon), which reformulates supervised contrastive learning as variational inference over latent class variables.<n>VarCon achieves state-of-the-art performance for contrastive learning frameworks, reaching 79.36% Top-1 accuracy on ImageNet-1K and 78.29% on CIFAR-100 with a ResNet-50 encoder.
  arXiv  Detail & Related papers  (2025-06-09T04:19:12Z)
• DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning [53.27049077100897]
  generative pre-training has been shown to yield discriminative representations, paving the way towards unified visual generation and understanding.<n>This work introduces self-conditioning, a mechanism that internally leverages the rich semantics inherent in denoising network to guide its own decoding layers.<n>Results are compelling: our method boosts both generation FID and recognition accuracy with 1% computational overhead and generalizes across diverse diffusion architectures.
  arXiv  Detail & Related papers  (2025-05-16T08:47:16Z)
• Conjuring Positive Pairs for Efficient Unification of Representation Learning and Image Synthesis [3.5900418884504095]
  Unified Self-Supervised Learning (SSL) methods have started to bridge the gap between representation learning and generative modeling.<n>Recent Unified SSL methods rely solely on semantic token reconstruction, which requires an external tokenizer during training.<n>We introduce Sorcen, a novel unified SSL framework, incorporating a synergic Contrastive-Reconstruction objective.
  arXiv  Detail & Related papers  (2025-03-19T09:53:11Z)
• VladVA: Discriminative Fine-tuning of LVLMs [67.14293827774827]
  Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the de facto approach for discriminative vision-language representation learning.<n>We propose to combine "the best of both worlds": a new training approach for discriminative fine-tuning of LVLMs.
  arXiv  Detail & Related papers  (2024-12-05T17:54:27Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation [70.2166826794421]
  We propose a differentiable geometric warping to conduct unsupervised data augmentation. We also propose a novel adversarial dual-student framework to improve the Mean-Teacher. Our solution significantly improves the performance and state-of-the-art results are achieved on both datasets.
  arXiv  Detail & Related papers  (2022-03-05T17:36:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.