IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models

• URL: http://arxiv.org/abs/2601.14188v1
• Date: Tue, 20 Jan 2026 17:45:24 GMT
• Title: IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models
• Abstract summary: In this work, we propose IIR-VLM, a VLM enhanced for In-context Instance-level Recognition.<n>We integrate pre-trained ILR expert models as auxiliary visual encoders to provide specialized features for learning diverse instances.<n>We validate IIR-VLM's efficacy on existing instance personalization benchmarks.
• Score: 31.791478473292184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Instance-level recognition (ILR) concerns distinguishing individual instances from one another, with person re-identification as a prominent example. Despite the impressive visual perception capabilities of modern VLMs, we find their performance on ILR unsatisfactory, often dramatically underperforming domain-specific ILR models. This limitation hinders many practical application of VLMs, e.g. where recognizing familiar people and objects is crucial for effective visual understanding. Existing solutions typically learn to recognize instances one at a time using instance-specific datasets, which not only incur substantial data collection and training costs but also struggle with fine-grained discrimination. In this work, we propose IIR-VLM, a VLM enhanced for In-context Instance-level Recognition. We integrate pre-trained ILR expert models as auxiliary visual encoders to provide specialized features for learning diverse instances, which enables VLMs to learn new instances in-context in a one-shot manner. Further, IIR-VLM leverages this knowledge for instance-aware visual understanding. We validate IIR-VLM's efficacy on existing instance personalization benchmarks. Finally, we demonstrate its superior ILR performance on a challenging new benchmark, which assesses ILR capabilities across varying difficulty and diverse categories, with person, face, pet and general objects as the instances at task.

Related papers

• Sign Language Recognition in the Age of LLMs [0.2609784101826761]
  Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks.<n>We investigate the capability of modern VLMs to perform isolated sign language recognition in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-04-13T09:26:16Z)
• Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models [19.361686225381447]
  Vision-language models (VLMs) are widely assumed to exhibit in-context learning (ICL)<n>We propose a new MM-ICL with Reasoning pipeline that augments each demonstration with a generated rationale alongside the answer.
  arXiv  Detail & Related papers  (2025-06-09T16:55:32Z)
• LLMs are Better Than You Think: Label-Guided In-Context Learning for Named Entity Recognition [10.920384665824807]
  In-context learning (ICL) enables large language models to perform new tasks using only a few demonstrations.<n>Existing ICL methods typically rely on task-agnostic semantic similarity for demonstration retrieval.<n>We introduce DEER, a training-free ICL approach that enables LLMs to make more informed entity predictions.
  arXiv  Detail & Related papers  (2025-05-29T17:54:32Z)
• Visual RAG: Expanding MLLM visual knowledge without fine-tuning [5.341192792319891]
  This paper introduces Visual RAG, that synergically combines the MLLMs capability to learn from the context, with a retrieval mechanism.<n>In this way, the resulting system is not limited to the knowledge extracted from the training data, but can be updated rapidly and easily without fine-tuning.<n>It greatly reduces the computational costs for improving the model image classification performance, and augments the model knowledge to new visual domains and tasks it was not trained for.
  arXiv  Detail & Related papers  (2025-01-18T17:43:05Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• AutoBench-V: Can Large Vision-Language Models Benchmark Themselves? [65.92331309449015]
  We introduce AutoBench-V, an automated framework for serving evaluation on demand, i.e., benchmarking LVLMs based on specific aspects of model capability.<n>Through an extensive evaluation of nine popular LVLMs across five demanded user inputs, the framework shows effectiveness and reliability.
  arXiv  Detail & Related papers  (2024-10-28T17:55:08Z)
• Large Vision-Language Models as Emotion Recognizers in Context Awareness [14.85890824622433]
  Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. This paper systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task.
  arXiv  Detail & Related papers  (2024-07-16T01:28:06Z)
• Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems [61.11799513362704]
  We propose learning an additional screening mechanism to identify discriminative clues commonly seen across instances and classes. We show that a common rationale detector can be learned by simply exploiting the GradCAM induced from the SSL objective.
  arXiv  Detail & Related papers  (2023-03-03T02:07:40Z)
• Learning Deep Representations via Contrastive Learning for Instance Retrieval [11.736450745549792]
  This paper makes the first attempt that tackles the problem using instance-discrimination based contrastive learning (CL) In this work, we approach this problem by exploring the capability of deriving discriminative representations from pre-trained and fine-tuned CL models.
  arXiv  Detail & Related papers  (2022-09-28T04:36:34Z)
• An Empirical Investigation of Representation Learning for Imitation [76.48784376425911]
  Recent work in vision, reinforcement learning, and NLP has shown that auxiliary representation learning objectives can reduce the need for large amounts of expensive, task-specific data. We propose a modular framework for constructing representation learning algorithms, then use our framework to evaluate the utility of representation learning for imitation.
  arXiv  Detail & Related papers  (2022-05-16T11:23:42Z)
• Revisiting Contrastive Learning for Few-Shot Classification [74.78397993160583]
  Instance discrimination based contrastive learning has emerged as a leading approach for self-supervised learning of visual representations. We show how one can incorporate supervision in the instance discrimination based contrastive self-supervised learning framework to learn representations that generalize better to novel tasks. We propose a novel model selection algorithm that can be used in conjunction with a universal embedding trained using CIDS to outperform state-of-the-art algorithms on the challenging Meta-Dataset benchmark.
  arXiv  Detail & Related papers  (2021-01-26T19:58:08Z)
• DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning [83.48587570246231]
  Visual Similarity plays an important role in many computer vision applications. Deep metric learning (DML) is a powerful framework for learning such similarities. We propose and study multiple complementary learning tasks, targeting conceptually different data relationships. We learn a single model to aggregate their training signals, resulting in strong generalization and state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-04-28T12:26:50Z)
• Exploit Clues from Views: Self-Supervised and Regularized Learning for Multiview Object Recognition [66.87417785210772]
  This work investigates the problem of multiview self-supervised learning (MV-SSL) A novel surrogate task for self-supervised learning is proposed by pursuing "object invariant" representation. Experiments shows that the recognition and retrieval results using view invariant prototype embedding (VISPE) outperform other self-supervised learning methods.
  arXiv  Detail & Related papers  (2020-03-28T07:06:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.