Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learners

• URL: http://arxiv.org/abs/2607.00125v1
• Date: Tue, 30 Jun 2026 20:00:50 GMT
• Title: Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learners
• Abstract summary: DeCoDe is a simple yet effective technique that enables off-the-shelf MLLMs to act as strong few-shot classifiers without any additional training.<n>Our approach builds on the idea of few-shot classification as a set of pairwise image comparisons, decomposing the task into a set of binary decisions.<n>Results show that the proposed simple decomposition technique can turn off-the-shelf MLLMs into powerful few-shot learners.
• Score: 32.73186456114494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable abilities when analyzing images, yet translating these capabilities to few-shot image classification remains challenging. To bridge this gap, we present DeCoDe, a simple yet effective technique that enables off-the-shelf MLLMs to act as strong few-shot classifiers without any additional training. Our approach builds on the idea of few-shot classification as a set of pairwise image comparisons, decomposing the task into a set of binary decisions. Given a query image and a support image from a candidate class, the MLLM is prompted to decide whether the two images depict the same class. The logit corresponding to an affirmative response is then used as a similarity score to assign the query image to the most likely class. While this already yields good results, we show that providing additional high-level information, such as the data domain, to the model further improves performance. Our evaluation provides an extensive analysis of various inference variants on a suite of twelve datasets, six established and six newly curated few-shot benchmarks spanning across diverse domains. The results show that the proposed simple decomposition technique can turn off-the-shelf MLLMs into powerful few-shot learners, significantly outperforming current state-of-the-art few-shot methods on both standard and novel domains. Code is available at https://github.com/yunhanwang1105/DeCoDe.

Related papers

• Indexing Multimodal Language Models for Large-scale Image Retrieval [15.19064519873711]
  Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored.<n>We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval.<n>Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines.
  arXiv  Detail & Related papers  (2026-04-14T19:59:36Z)
• More Images, More Problems? A Controlled Analysis of VLM Failure Modes [80.64323947730905]
  Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored.<n>We introduce MIMIC, a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs.
  arXiv  Detail & Related papers  (2026-01-12T18:45:13Z)
• Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models [4.499940819352075]
  Large Vision-Language Models (LVLMs) have demonstrated impressive performance on vision-language reasoning tasks.<n>We present a novel method that transforms zero-shot fine-grained image classification into a visual question-answering framework.<n>Our proposed method consistently outperforms the current state-of-the-art (SOTA) approach.
  arXiv  Detail & Related papers  (2025-10-04T18:56:41Z)
• Chain-of-Thought Re-ranking for Image Retrieval Tasks [16.13448876168839]
  We propose a novel Chain-of-Thought Re-Ranking (CoTRR) method to address image retrieval.<n>By allowing MLLM to perform listwise reasoning, our method supports global comparison, consistent reasoning, and interpretable decision-making.<n>Our method achieves state-of-the-art performance across three image retrieval tasks, including text-to-image retrieval (TIR), composed image retrieval (CIR) and chat-based image retrieval (Chat-IR)
  arXiv  Detail & Related papers  (2025-09-18T08:48:46Z)
• MIBench: Evaluating Multimodal Large Language Models over Multiple Images [70.44423964171088]
  We propose a new benchmark MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in multi-image scenarios. Specifically, MIBench categorizes the multi-image abilities into three scenarios: multi-image instruction (MII), multimodal knowledge-seeking (MKS) and multimodal in-context learning (MIC) The results reveal that although current models excel in single-image tasks, they exhibit significant shortcomings when faced with multi-image inputs.
  arXiv  Detail & Related papers  (2024-07-21T21:22:58Z)
• What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models [11.683093317651517]
  Large language models (LLMs) have been effectively used for many computer vision tasks, including image classification.<n>We present a simple yet effective approach for zero-shot image classification using multimodal LLMs.<n>Our results demonstrate its remarkable effectiveness, surpassing benchmark accuracy on multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T16:05:15Z)
• Advancing Image Retrieval with Few-Shot Learning and Relevance Feedback [5.770351255180495]
  Image Retrieval with Relevance Feedback (IRRF) involves iterative human interaction during the retrieval process. We propose a new scheme based on a hyper-network, that is tailored to the task and facilitates swift adjustment to user feedback. We show that our method can attain SoTA results in few-shot one-class classification and reach comparable results in binary classification task of few-shot open-set recognition.
  arXiv  Detail & Related papers  (2023-12-18T10:20:28Z)
• CLAMP: Contrastive LAnguage Model Prompt-tuning [89.96914454453791]
  We show that large language models can achieve good image classification performance when adapted this way. Our approach beats state-of-the-art mLLMs by 13% and slightly outperforms contrastive learning with a custom text model.
  arXiv  Detail & Related papers  (2023-12-04T05:13:59Z)
• Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs [49.88461345825586]
  This paper proposes a new framework to enhance the fine-grained image understanding abilities of MLLMs. We present a new method for constructing the instruction tuning dataset at a low cost by leveraging annotations in existing datasets. We show that our model exhibits a 5.2% accuracy improvement over Qwen-VL and surpasses the accuracy of Kosmos-2 by 24.7%.
  arXiv  Detail & Related papers  (2023-10-01T05:53:15Z)
• Self-Supervised Open-Ended Classification with Small Visual Language Models [60.23212389067007]
  We present Self-Context Adaptation (SeCAt), a self-supervised approach that unlocks few-shot abilities for open-ended classification with small visual language models. By using models with approximately 1B parameters we outperform the few-shot abilities of much larger models, such as Frozen and FROMAGe.
  arXiv  Detail & Related papers  (2023-09-30T21:41:21Z)
• Cap2Aug: Caption guided Image to Image data Augmentation [41.53127698828463]
  Cap2Aug is an image-to-image diffusion model-based data augmentation strategy using image captions as text prompts. We generate captions from the limited training images and using these captions edit the training images using an image-to-image stable diffusion model. This strategy generates augmented versions of images similar to the training images yet provides semantic diversity across the samples.
  arXiv  Detail & Related papers  (2022-12-11T04:37:43Z)
• Few-Shot Learning with Part Discovery and Augmentation from Unlabeled Images [79.34600869202373]
  We show that inductive bias can be learned from a flat collection of unlabeled images, and instantiated as transferable representations among seen and unseen classes. Specifically, we propose a novel part-based self-supervised representation learning scheme to learn transferable representations. Our method yields impressive results, outperforming the previous best unsupervised methods by 7.74% and 9.24%.
  arXiv  Detail & Related papers  (2021-05-25T12:22:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.