Knowledge Transfer Across Modalities with Natural Language Supervision

• URL: http://arxiv.org/abs/2411.15611v1
• Date: Sat, 23 Nov 2024 17:26:50 GMT
• Title: Knowledge Transfer Across Modalities with Natural Language Supervision
• Authors: Carlo Alberto Barbano, Luca Molinaro, Emanuele Aiello, Marco Grangetto,
• Abstract summary: We present a way to learn novel concepts by only using their textual description. Similarly to human perception, we leverage cross-modal interaction to introduce new concepts. We show that Knowledge Transfer can successfully introduce novel concepts in multimodal models, in a very efficient manner.
• Score: 8.493435472659646
• License:
• Abstract: We present a way to learn novel concepts by only using their textual description. We call this method Knowledge Transfer. Similarly to human perception, we leverage cross-modal interaction to introduce new concepts. We hypothesize that in a pre-trained visual encoder there are enough low-level features already learned (e.g. shape, appearance, color) that can be used to describe previously unknown high-level concepts. Provided with a textual description of the novel concept, our method works by aligning the known low-level features of the visual encoder to its high-level textual description. We show that Knowledge Transfer can successfully introduce novel concepts in multimodal models, in a very efficient manner, by only requiring a single description of the target concept. Our approach is compatible with both separate textual and visual encoders (e.g. CLIP) and shared parameters across modalities. We also show that, following the same principle, Knowledge Transfer can improve concepts already known by the model. Leveraging Knowledge Transfer we improve zero-shot performance across different tasks such as classification, segmentation, image-text retrieval, and captioning.

Related papers

• Scaling Concept With Text-Guided Diffusion Models [53.80799139331966]
  Instead of replacing a concept, can we enhance or suppress the concept itself? We introduce ScalingConcept, a simple yet effective method to scale decomposed concepts up or down in real input without introducing new elements. More importantly, ScalingConcept enables a variety of novel zero-shot applications across image and audio domains.
  arXiv  Detail & Related papers  (2024-10-31T17:09:55Z)
• Explainable Concept Generation through Vision-Language Preference Learning [7.736445799116692]
  Concept-based explanations have become a popular choice for explaining deep neural networks post-hoc. We devise a reinforcement learning-based preference optimization algorithm that fine-tunes the vision-language generative model. In addition to showing the efficacy and reliability of our method, we show how our method can be used as a diagnostic tool for analyzing neural networks.
  arXiv  Detail & Related papers  (2024-08-24T02:26:42Z)
• Conceptual Codebook Learning for Vision-Language Models [27.68834532978939]
  We propose Codebook Learning (CoCoLe) to address the challenge of improving the generalization capability of vision-language models (VLMs) We learn a conceptual codebook consisting of visual concepts as keys and conceptual prompts as values. We observe that this conceptual codebook learning method is able to achieve enhanced alignment between visual and linguistic modalities.
  arXiv  Detail & Related papers  (2024-07-02T15:16:06Z)
• Learning Structure and Knowledge Aware Representation with Large Language Models for Concept Recommendation [50.31872005772817]
  Concept recommendation aims to suggest the next concept for learners to study based on their knowledge states and the human knowledge system. Previous approaches have not effectively integrated the human knowledge system into the process of designing these educational models. We propose a novel Structure and Knowledge Aware Representation learning framework for concept Recommendation (SKarREC)
  arXiv  Detail & Related papers  (2024-05-21T01:35:36Z)
• Concept-Guided Prompt Learning for Generalization in Vision-Language Models [33.361744437967126]
  We propose Concept-Guided Prompt Learning for vision-language models. We leverage the well-learned knowledge of Contrastive Language-Image Pretraining to create a visual concept cache. In order to refine the text features, we develop a projector that transforms multi-level visual features into text features.
  arXiv  Detail & Related papers  (2024-01-15T04:04:47Z)
• Language-Informed Visual Concept Learning [22.911347501969857]
  We train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes. We then anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model. At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts.
  arXiv  Detail & Related papers  (2023-12-06T16:24:47Z)
• Create Your World: Lifelong Text-to-Image Diffusion [75.14353789007902]
  We propose Lifelong text-to-image Diffusion Model (L2DM) to overcome knowledge "catastrophic forgetting" for the past encountered concepts. In respect of knowledge "catastrophic forgetting", our L2DM framework devises a task-aware memory enhancement module and a elastic-concept distillation module. Our model can generate more faithful image across a range of continual text prompts in terms of both qualitative and quantitative metrics.
  arXiv  Detail & Related papers  (2023-09-08T16:45:56Z)
• K-LITE: Learning Transferable Visual Models with External Knowledge [242.3887854728843]
  K-LITE (Knowledge-augmented Language-Image Training and Evaluation) is a strategy to leverage external knowledge to build transferable visual systems. In training, it enriches entities in natural language with WordNet and Wiktionary knowledge. In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts.
  arXiv  Detail & Related papers  (2022-04-20T04:47:01Z)
• FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations [99.54048050189971]
  We present a framework for learning new visual concepts quickly, guided by multiple naturally occurring data streams. The learned concepts support downstream applications, such as answering questions by reasoning about unseen images. We demonstrate the effectiveness of our model on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-03-30T19:45:00Z)
• Toward a Visual Concept Vocabulary for GAN Latent Space [74.12447538049537]
  This paper introduces a new method for building open-ended vocabularies of primitive visual concepts represented in a GAN's latent space. Our approach is built from three components: automatic identification of perceptually salient directions based on their layer selectivity; human annotation of these directions with free-form, compositional natural language descriptions. Experiments show that concepts learned with our approach are reliable and composable -- generalizing across classes, contexts, and observers.
  arXiv  Detail & Related papers  (2021-10-08T17:58:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.