Show Me Examples: Inferring Visual Concepts from Image Sets

• URL: http://arxiv.org/abs/2607.02402v2
• Date: Mon, 06 Jul 2026 17:05:13 GMT
• Title: Show Me Examples: Inferring Visual Concepts from Image Sets
• Abstract summary: We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability.<n>Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept.<n>We propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries.
• Score: 22.395558604172127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task, often ignoring the visual context or defaulting to biased generations. To address this gap, we propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries. Experiments on synthetic data and large-scale ImageNet/WordNet data show that our model generates more accurate and diverse outputs and generalizes to unseen concepts and modalities such as sketches.

Related papers

• Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders [9.716523835964045]
  We propose a framework based on Sparse Autoencoders (SAEs) to extract and analyze visual, textual, and multimodal concepts from Vision Language Models (VLMs)<n>Our framework improves visual concept quality by up to 45% compared to existing SAE-based methods.
  arXiv  Detail & Related papers  (2026-06-19T08:08:43Z)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees [59.16312652369709]
  Contrastive Language-Image Pre-training (CLIP)citepradford2021learning has emerged as a pivotal model in computer vision and multimodal learning.<n>CLIP struggles with potential information misalignment in many image-text datasets and suffers from entangled representation.<n>We introduce ours, a novel approach that identifies and aligns the most relevant visual and textual representations in a modular manner.
  arXiv  Detail & Related papers  (2025-07-29T22:26:20Z)
• MyVLM: Personalizing VLMs for User-Specific Queries [78.33252556805931]
  We take a first step toward the personalization of vision-language models, enabling them to learn and reason over user-provided concepts. To effectively recognize a variety of user-specific concepts, we augment the VLM with external concept heads that function as toggles for the model. Having recognized the concept, we learn a new concept embedding in the intermediate feature space of the VLM. This embedding is tasked with guiding the language model to naturally integrate the target concept in its generated response.
  arXiv  Detail & Related papers  (2024-03-21T17:51:01Z)
• Visual Concept-driven Image Generation with Text-to-Image Diffusion Model [65.96212844602866]
  Text-to-image (TTI) models have demonstrated impressive results in generating high-resolution images of complex scenes.<n>Recent approaches have extended these methods with personalization techniques that allow them to integrate user-illustrated concepts.<n>However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive.<n>We propose a concept-driven TTI personalization framework that addresses these core challenges.
  arXiv  Detail & Related papers  (2024-02-18T07:28:37Z)
• Language-Informed Visual Concept Learning [22.911347501969857]
  We train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes. We then anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model. At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts.
  arXiv  Detail & Related papers  (2023-12-06T16:24:47Z)
• Context Diffusion: In-Context Aware Image Generation [27.63447546705641]
  Context Diffusion is a diffusion-based framework that enables image generation models to learn from visual examples presented in context.<n>Our experiments and human evaluation demonstrate that Context Diffusion excels in both in-domain and out-of-domain tasks.
  arXiv  Detail & Related papers  (2023-12-06T16:19:51Z)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists [66.85125112199898]
  We develop a unified language interface for computer vision tasks that abstracts away task-specific design choices. Our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models.
  arXiv  Detail & Related papers  (2023-09-30T14:26:43Z)
• Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Models [80.75258849913574]
  In this paper, we consider the inverse problem -- given a collection of different images, can we discover the generative concepts that represent each image? We present an unsupervised approach to discover generative concepts from a collection of images, disentangling different art styles in paintings, objects, and lighting from kitchen scenes, and discovering image classes given ImageNet images.
  arXiv  Detail & Related papers  (2023-06-08T17:02:15Z)
• ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models [79.10890337599166]
  We introduce ConceptBed, a large-scale dataset that consists of 284 unique visual concepts and 33K composite text prompts. We evaluate visual concepts that are either objects, attributes, or styles, and also evaluate four dimensions of compositionality: counting, attributes, relations, and actions. Our results point to a trade-off between learning the concepts and preserving the compositionality which existing approaches struggle to overcome.
  arXiv  Detail & Related papers  (2023-06-07T18:00:38Z)
• Concept Generalization in Visual Representation Learning [39.32868843527767]
  We argue that semantic relationships between seen and unseen concepts affect generalization performance. We propose ImageNet-CoG, a novel benchmark on the ImageNet dataset that enables measuring concept generalization in a principled way.
  arXiv  Detail & Related papers  (2020-12-10T13:13:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.