G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

• URL: http://arxiv.org/abs/2608.26744v1
• Date: Thu, 27 Aug 2026 07:34:07 GMT
• Title: G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification
• Abstract summary: G2D is a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.<n>It achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.<n>It also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning.
• Score: 14.607727417863273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Zero-shot classification needs efficient label retrieval and fine-grained visual reasoning, yet discriminative and generative vision-language models fail in complementary ways.When CLIP's top-1 prediction is wrong, the correct label often remains in its top-$K$ shortlist, making disambiguation rather than recall the key challenge.Standalone generative models, however, are hindered by large label spaces and unconstrained outputs.This complementarity motivates separating broad candidate retrieval from fine-grained, image-grounded verification.We propose G2D, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.Candidate names and CLIP probabilities provide a structured prior for resolving visually similar classes.Fixed confidence routing, entropy-adaptive candidate sizing, and trie-constrained decoding focus generative reasoning on uncertain samples and ensure one valid output for each input at test time.Across eight benchmarks, G2D achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.Across seven generator configurations, candidate-set verification improves average accuracy by 1.08--27.42 percentage points.G2D also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning. Code: https://github.com/Harzva/G2D

Related papers

• GATE: Reliability-Gated Gaussian Evidence Fusion for Training-Free Test-Time Adaptation of Vision-Language Models [5.025577782616721]
  GATE is a training-free two-pass transductive test-time adaptation framework.<n>It uses the unlabeled target set while keeping the image encoder, text encoder, and prompt parameters fully frozen.<n>It improves zero-shot performance by an average of 5.41 points and outperforms the strongest non-GATE baseline by 1.94 points.
  arXiv  Detail & Related papers  (2026-08-29T18:33:23Z)
• Visual Distribution Anchoring for Efficient Prompt Tuning [0.6918455480131249]
  We propose VDA (Visual Distribution Anchoring), a training-free target adaptation framework.<n>It augments a frozen semantic classifier with class-level visual prototypes estimated offline from an unlabeled target pool.<n>We show that class-specific partitioning drives gains and that visually local pseudo-label errors can remain useful despite being class-incorrect.
  arXiv  Detail & Related papers  (2026-07-31T02:43:07Z)
• CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification [65.46685389276443]
  We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts.<n>We then formulate purification risk as the KL divergence between the joint distributions purification process.<n>We propose two variants for our CLIPure approach: CLI-Diff which models the likelihood of images' latent vectors, and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and a photo of a.''
  arXiv  Detail & Related papers  (2025-02-25T13:09:34Z)
• African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification [53.89380284760555]
  textttFOCI (textbfFine-grained textbfObject textbfClasstextbfIfication) is a difficult multiple-choice benchmark for fine-grained object classification. textttFOCIxspace complements five popular classification datasets with four domain-specific subsets from ImageNet-21k.
  arXiv  Detail & Related papers  (2024-06-20T16:59:39Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation [20.57370550156505]
  ReCLIP is a source-free domain adaptation method for vision-language models. We demonstrate ReCLIP reduces the average error rate of CLIP from 30.17% to 25.06% on 22 image classification benchmarks.
  arXiv  Detail & Related papers  (2023-08-04T18:11:40Z)
• Masked Unsupervised Self-training for Zero-shot Image Classification [98.23094305347709]
  Masked Unsupervised Self-Training (MUST) is a new approach which leverages two different and complimentary sources of supervision: pseudo-labels and raw images. MUST improves upon CLIP by a large margin and narrows the performance gap between unsupervised and supervised classification.
  arXiv  Detail & Related papers  (2022-06-07T02:03:06Z)
• Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density [125.64297244986552]
  We propose an unsupervised validation criterion that measures the density of soft neighborhoods by computing the entropy of the similarity distribution between points. Our criterion is simpler than competing validation methods, yet more effective.
  arXiv  Detail & Related papers  (2021-08-24T17:41:45Z)
• Self-Supervised Classification Network [3.8073142980733]
  Self-supervised end-to-end classification neural network learns labels and representations simultaneously. First unsupervised end-to-end classification network to perform well on the large-scale ImageNet dataset.
  arXiv  Detail & Related papers  (2021-03-19T19:29:42Z)
• Joint Visual and Temporal Consistency for Unsupervised Domain Adaptive Person Re-Identification [64.37745443119942]
  This paper jointly enforces visual and temporal consistency in the combination of a local one-hot classification and a global multi-class classification. Experimental results on three large-scale ReID datasets demonstrate the superiority of proposed method in both unsupervised and unsupervised domain adaptive ReID tasks.
  arXiv  Detail & Related papers  (2020-07-21T14:31:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.