Bridging the gap to real-world language-grounded visual concept learning

• URL: http://arxiv.org/abs/2510.21412v2
• Date: Tue, 28 Oct 2025 05:32:23 GMT
• Title: Bridging the gap to real-world language-grounded visual concept learning
• Authors: Whie Jung, Semin Kim, Junee Kim, Seunghoon Hong,
• Abstract summary: We propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes.<n>We demonstrate the effectiveness of our framework on subsets of ImageNet, CelebA-HQ, and AFHQ.
• Score: 18.37183228394834
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human intelligence effortlessly interprets visual scenes along a rich spectrum of semantic dimensions. However, existing approaches to language-grounded visual concept learning are limited to a few predefined primitive axes, such as color and shape, and are typically explored in synthetic datasets. In this work, we propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes. Leveraging a pretrained vision-language model and our universal prompting strategy, our framework identifies a diverse image-related axes without any prior knowledge. Our universal concept encoder adaptively binds visual features to the discovered axes without introducing additional model parameters for each concept. To ground visual concepts along the discovered axes, we optimize a compositional anchoring objective, which ensures that each axis can be independently manipulated without affecting others. We demonstrate the effectiveness of our framework on subsets of ImageNet, CelebA-HQ, and AFHQ, showcasing superior editing capabilities across diverse real-world concepts that are too varied to be manually predefined. Our method also exhibits strong compositional generalization, outperforming existing visual concept learning and text-based editing methods. The code is available at https://github.com/whieya/Language-grounded-VCL.

Related papers

• Insight: Interpretable Semantic Hierarchies in Vision-Language Encoders [52.94006363830628]
  Language-aligned vision foundation models perform strongly across diverse downstream tasks.<n>Recent works decompose these representations into human-interpretable concepts, but provide poor spatial grounding and are limited to image classification tasks.<n>We propose Insight, a language-aligned concept foundation model that provides fine-grained concepts, which are human-interpretable and spatially grounded in the input image.
  arXiv  Detail & Related papers  (2026-01-20T09:57:26Z)
• Perception Encoder: The best visual embeddings are not at the output of the network [70.86738083862099]
  We introduce Perception (PE), a vision encoder for image and video understanding trained via simple vision-language learning.<n>We find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks.<n>Together, our PE family of models achieves best-in-class results on a wide variety of tasks.
  arXiv  Detail & Related papers  (2025-04-17T17:59:57Z)
• Self-Evolving Visual Concept Library using Vision-Language Critics [38.15146001218907]
  Building effective visual concept libraries is challenging, as manual definition is labor-intensive.<n>Our approach, ESCHER, takes a library learning perspective to iteratively discover and improve visual concepts.<n>We empirically demonstrate the ability of ESCHER to learn a concept library for zero-shot, few-shot, and fine-tuning visual classification tasks.
  arXiv  Detail & Related papers  (2025-03-31T19:47:55Z)
• Compositional Entailment Learning for Hyperbolic Vision-Language Models [54.41927525264365]
  We show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs.<n>We propose Compositional Entailment Learning for hyperbolic vision-language models.<n> Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning.
  arXiv  Detail & Related papers  (2024-10-09T14:12:50Z)
• Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations [7.736445799116692]
  Concept-based methods have become a popular choice for explaining deep neural networks post-hoc.<n>We devise a reinforcement learning-based preference optimization algorithm that fine-tunes a vision-language generative model.<n>We demonstrate our method's ability to efficiently and reliably articulate diverse concepts.
  arXiv  Detail & Related papers  (2024-08-24T02:26:42Z)
• Pre-trained Vision-Language Models Learn Discoverable Visual Concepts [33.302556000017844]
  We aim to answer this question as visual concepts learned "for free" would enable wide applications.<n>We assume that the visual concepts, if captured by pre-trained VLMs, can be extracted by their vision-language interface with text-based concept prompts.<n>Our proposed concept discovery and learning framework is thus designed to identify a diverse list of generic visual concepts.
  arXiv  Detail & Related papers  (2024-04-19T06:41:32Z)
• Language-Informed Visual Concept Learning [22.911347501969857]
  We train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes. We then anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model. At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts.
  arXiv  Detail & Related papers  (2023-12-06T16:24:47Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)
• K-LITE: Learning Transferable Visual Models with External Knowledge [242.3887854728843]
  K-LITE (Knowledge-augmented Language-Image Training and Evaluation) is a strategy to leverage external knowledge to build transferable visual systems. In training, it enriches entities in natural language with WordNet and Wiktionary knowledge. In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts.
  arXiv  Detail & Related papers  (2022-04-20T04:47:01Z)
• FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations [99.54048050189971]
  We present a framework for learning new visual concepts quickly, guided by multiple naturally occurring data streams. The learned concepts support downstream applications, such as answering questions by reasoning about unseen images. We demonstrate the effectiveness of our model on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-03-30T19:45:00Z)
• Unsupervised Learning of Compositional Energy Concepts [70.11673173291426]
  We propose COMET, which discovers and represents concepts as separate energy functions. Comet represents both global concepts as well as objects under a unified framework.
  arXiv  Detail & Related papers  (2021-11-04T17:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.