A Concept-Centric Approach to Multi-Modality Learning

• URL: http://arxiv.org/abs/2412.13847v1
• Date: Wed, 18 Dec 2024 13:40:21 GMT
• Title: A Concept-Centric Approach to Multi-Modality Learning
• Authors: Yuchong Geng, Ao Tang,
• Abstract summary: We introduce a new multi-modality learning framework to create a more efficient AI system. Our framework achieves on par with benchmark models while demonstrating more efficient learning curves.
• Score: 3.828996378105142
• License:
• Abstract: In an effort to create a more efficient AI system, we introduce a new multi-modality learning framework that leverages a modality-agnostic concept space possessing abstract knowledge and a set of modality-specific projection models tailored to process distinct modality inputs and map them onto the concept space. Decoupled from specific modalities and their associated projection models, the concept space focuses on learning abstract knowledge that is universally applicable across modalities. Subsequently, the knowledge embedded into the concept space streamlines the learning processes of modality-specific projection models. We evaluate our framework on two popular tasks: Image-Text Matching and Visual Question Answering. Our framework achieves performance on par with benchmark models while demonstrating more efficient learning curves.

Related papers

• Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations? [55.99654128127689]
  Cross-modal contrastive distillation has recently been explored for learning effective 3D representations. Existing methods focus primarily on modality-shared features, neglecting the modality-specific features during the pre-training process. We propose a new framework, namely CMCR, to address these shortcomings.
  arXiv  Detail & Related papers  (2024-12-12T06:09:49Z)
• InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillation [18.793275018467163]
  This paper presents InFiConD, a novel framework that leverages visual concepts to implement the knowledge distillation process. We develop a novel knowledge distillation pipeline based on extracting text-aligned visual concepts from a concept corpus. InFiConD's interface allows users to interactively fine-tune the student model by manipulating concept influences directly in the user interface.
  arXiv  Detail & Related papers  (2024-06-25T16:56:45Z)
• Mining Frequent Structures in Conceptual Models [2.625701175074974]
  We propose a general approach to the problem of discovering frequent structures in conceptual models. We implement our approach by focusing on two widely-used conceptual modeling languages. This tool can be used to identify both effective and ineffective modeling practices.
  arXiv  Detail & Related papers  (2024-06-11T10:24:02Z)
• Visual Concept-driven Image Generation with Text-to-Image Diffusion Model [65.96212844602866]
  Text-to-image (TTI) models have demonstrated impressive results in generating high-resolution images of complex scenes. Recent approaches have extended these methods with personalization techniques that allow them to integrate user-illustrated concepts. However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive. We propose a concept-driven TTI personalization framework that addresses these core challenges.
  arXiv  Detail & Related papers  (2024-02-18T07:28:37Z)
• Advancing Ante-Hoc Explainable Models through Generative Adversarial Networks [24.45212348373868]
  This paper presents a novel concept learning framework for enhancing model interpretability and performance in visual classification tasks. Our approach appends an unsupervised explanation generator to the primary classifier network and makes use of adversarial training. This work presents a significant step towards building inherently interpretable deep vision models with task-aligned concept representations.
  arXiv  Detail & Related papers  (2024-01-09T16:16:16Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Contrastive Language-Image Pre-Training with Knowledge Graphs [33.211811772961234]
  We propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model. Our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities.
  arXiv  Detail & Related papers  (2022-10-17T09:49:22Z)
• A Unified Continuous Learning Framework for Multi-modal Knowledge Discovery and Pre-training [73.7507857547549]
  We propose to unify knowledge discovery and multi-modal pre-training in a continuous learning framework. For knowledge discovery, a pre-trained model is used to identify cross-modal links on a graph. For model pre-training, the knowledge graph is used as the external knowledge to guide the model updating.
  arXiv  Detail & Related papers  (2022-06-11T16:05:06Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)
• A Competence-aware Curriculum for Visual Concepts Learning via Question Answering [95.35905804211698]
  We propose a competence-aware curriculum for visual concept learning in a question-answering manner. We design a neural-symbolic concept learner for learning the visual concepts and a multi-dimensional Item Response Theory (mIRT) model for guiding the learning process. Experimental results on CLEVR show that with a competence-aware curriculum, the proposed method achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2020-07-03T05:08:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.