The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models

• URL: http://arxiv.org/abs/2608.23634v1
• Date: Sun, 23 Aug 2026 15:02:33 GMT
• Title: The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models
• Abstract summary: Many few-shot adaptation methods classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features.<n>We ask what is the right ratio, can it be estimated without validation data, and is finding it where the performance is?<n>Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity.<n> validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average.
• Score: 22.461983628330277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many few-shot adaptation methods for vision-language models classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features, with a single blending ratio routinely tuned on held-out labels, often on the test set itself. We ask what the family's own bias-variance justification invites: what is the right ratio, can it be estimated without validation data, and is finding it where the performance is? First, the ratio minimising prototype mean-squared error has a closed form whose support-set plug-in is exactly a positive-part James-Stein coefficient shrinking towards the text prototype. Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity: on the 950 primary-tier cells where it is defined it trails a test-set-oracle ratio by 8.5 points. It saturates near 1, discarding the text prior for a nearest-class-mean classifier, because 78% of the text-image prototype distance it treats as bias is a class-independent offset that the arg max largely cancels. We prove the mechanism and bound its share of the damage at 26% by a counterfactual. Second, leave-one-out on the support set alone sets a ratio landing within 0.9 points of the oracle blend, so it is estimable without validation data. Third, validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average, and at K >= 4 all four validation-free baselines sit above the oracle, the linear probes by margins excluding zero. These results locate the ceiling in the model class, not the hyperparameter: the ratio can be set near-optimally for free, and it is still not where the performance is. Code, cached features, per-cell records: https://huggingface.co/datasets/Liangzhi-Li/clipbench-blending

Related papers

• When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification [0.0]
  Class-imbalance handling is routinely evaluated on a single benchmark dataset.<n>Under a leakage-free nested crossvalidation protocol, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021.<n> Threshold-tuning benefit is non-monotonic in the imbalance ratio.
  arXiv  Detail & Related papers  (2026-08-17T05:58:33Z)
• Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel [0.0]
  We study a fixed panel of sixteen lightweight persona-conditioned GPT-4.1 configurations in repeated strategic games.<n>Variation was strongly prompt-indexed, but its share depended on uncertainty assumptions.<n>Results concern one fixed model-prompt panel and do not establish human substitutability.
  arXiv  Detail & Related papers  (2026-08-02T04:10:11Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness [0.0]
  Few-shot prompting, the practice of prepending a small number of input-output demonstration pairs to a query before presenting it to a large language model, is widely adopted in NLP.<n>Yet little systematic work investigates how shot count interacts with model scale, architecture, and output format compliance in classification performance.
  arXiv  Detail & Related papers  (2026-07-25T00:31:40Z)
• K-ABENA: K-Adaptive Backpropagation with Error-based N-exclusion Algorithm : (Compensated Loss-Based Sample Exclusion with Unbiased Gradient Estimation) [0.0]
  K-ABENAK-Adaptive Backpropagation with Error-based N-exclusion is a selective computation framework.<n>It reduces per-it computation training cost by excluding a fraction of low-loss ("minor") observations.<n>It collapses to 0.386 accuracy under label limitation (baseline: 0.832) and to 0.53 AUC under extreme imbalance.
  arXiv  Detail & Related papers  (2026-07-07T06:58:51Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization [54.749573452394664]
  We find that accuracy collapses as instance data grows, even when the formulation itself is simple.<n>We address this via a simple inference-time approach, BIND, which externalizes numeric data to structured files so the model binds data rather than from prompt prompt.<n>We validate our hypothesis by finetuning a model exclusively on binding and show that it outperforms end-to-end SFT and RL across three structurally distinct optimization categories.
  arXiv  Detail & Related papers  (2026-05-20T21:25:41Z)
• Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness [61.45587642780908]
  We propose a three-step approach for parameter-efficient fine-tuning of image-text foundation models.<n>Our method improves its two key components: minority samples identification and the robust training algorithm.<n>Our theoretical analysis shows that our PPA enhances minority group identification and is Bayes optimal for minimizing the balanced group error.
  arXiv  Detail & Related papers  (2025-03-12T15:46:12Z)
• CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification [65.46685389276443]
  We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts.<n>We then formulate purification risk as the KL divergence between the joint distributions purification process.<n>We propose two variants for our CLIPure approach: CLI-Diff which models the likelihood of images' latent vectors, and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and a photo of a.''
  arXiv  Detail & Related papers  (2025-02-25T13:09:34Z)
• FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning [46.95063831057502]
  We propose emphFreeMatch to define and adjust the confidence threshold in a self-adaptive manner according to the model's learning status. FreeMatch achieves textbf5.78%, textbf13.59%, and textbf1.28% error rate reduction over the latest state-of-the-art method FlexMatch on CIFAR-10 with 1 label per class.
  arXiv  Detail & Related papers  (2022-05-15T10:07:52Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.