Cross-Modal Alignment Learning of Vision-Language Conceptual Systems

• URL: http://arxiv.org/abs/2208.01744v1
• Date: Sun, 31 Jul 2022 08:39:53 GMT
• Title: Cross-Modal Alignment Learning of Vision-Language Conceptual Systems
• Authors: Taehyeong Kim, Hyeonseop Song, Byoung-Tak Zhang
• Abstract summary: We propose methods for learning aligned vision-language conceptual systems inspired by infants' word learning mechanisms. The proposed model learns the associations of visual objects and words online and gradually constructs cross-modal relational graph networks.
• Score: 24.423011687551433
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human infants learn the names of objects and develop their own conceptual systems without explicit supervision. In this study, we propose methods for learning aligned vision-language conceptual systems inspired by infants' word learning mechanisms. The proposed model learns the associations of visual objects and words online and gradually constructs cross-modal relational graph networks. Additionally, we also propose an aligned cross-modal representation learning method that learns semantic representations of visual objects and words in a self-supervised manner based on the cross-modal relational graph networks. It allows entities of different modalities with conceptually the same meaning to have similar semantic representation vectors. We quantitatively and qualitatively evaluate our method, including object-to-word mapping and zero-shot learning tasks, showing that the proposed model significantly outperforms the baselines and that each conceptual system is topologically aligned.

Related papers

• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Grounded learning for compositional vector semantics [1.4344589271451351]
  This work proposes a way for compositional distributional semantics to be implemented within a spiking neural network architecture. We also describe a means of training word representations using labelled images.
  arXiv  Detail & Related papers  (2024-01-10T22:12:34Z)
• Advancing Ante-Hoc Explainable Models through Generative Adversarial Networks [24.45212348373868]
  This paper presents a novel concept learning framework for enhancing model interpretability and performance in visual classification tasks. Our approach appends an unsupervised explanation generator to the primary classifier network and makes use of adversarial training. This work presents a significant step towards building inherently interpretable deep vision models with task-aligned concept representations.
  arXiv  Detail & Related papers  (2024-01-09T16:16:16Z)
• CLiC: Concept Learning in Context [54.81654147248919]
  This paper builds upon recent advancements in visual concept learning. It involves acquiring a visual concept from a source image and subsequently applying it to an object in a target image. To localize the concept learning, we employ soft masks that contain both the concept within the mask and the surrounding image area.
  arXiv  Detail & Related papers  (2023-11-28T01:33:18Z)
• Perceptual Grouping in Contrastive Vision-Language Models [59.1542019031645]
  We show how vision-language models are able to understand where objects reside within an image and group together visually related parts of the imagery. We propose a minimal set of modifications that results in models that uniquely learn both semantic and spatial information.
  arXiv  Detail & Related papers  (2022-10-18T17:01:35Z)
• Pretraining on Interactions for Learning Grounded Affordance Representations [22.290431852705662]
  We train a neural network to predict objects' trajectories in a simulated interaction. We show that our network's latent representations differentiate between both observed and unobserved affordances. Our results suggest a way in which modern deep learning approaches to grounded language learning can be integrated with traditional formal semantic notions of lexical representations.
  arXiv  Detail & Related papers  (2022-07-05T19:19:53Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)
• Building a visual semantics aware object hierarchy [0.0]
  We propose a novel unsupervised method to build visual semantics aware object hierarchy. Our intuition in this paper comes from real-world knowledge representation where concepts are hierarchically organized. The evaluation consists of two parts, firstly we apply the constructed hierarchy on the object recognition task and then we compare our visual hierarchy and existing lexical hierarchies to show the validity of our method.
  arXiv  Detail & Related papers  (2022-02-26T00:10:21Z)
• Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning [3.441021278275805]
  We design a two-stream model for grounding language learning in vision. The model first learns to align visual and language representations with the MS COCO dataset. After training, the language stream of this model is a stand-alone language model capable of embedding concepts in a visually grounded semantic space.
  arXiv  Detail & Related papers  (2021-11-13T19:54:15Z)
• Cross-Modal Discrete Representation Learning [73.68393416984618]
  We present a self-supervised learning framework that learns a representation that captures finer levels of granularity across different modalities. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities.
  arXiv  Detail & Related papers  (2021-06-10T00:23:33Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.