To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models

• URL: http://arxiv.org/abs/2406.20054v1
• Date: Fri, 28 Jun 2024 17:07:06 GMT
• Title: To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models
• Authors: Bastien Liétard, Pascal Denis, Mikaella Keller,
• Abstract summary: Polysemy and synonymy are crucial facets of lexical ambiguity. In this paper, we introduce Concept Induction, the unsupervised task of learning a soft clustering among words. We propose a bi-level approach to Concept Induction that leverages both a local lemma-centric view and a global cross-lexicon perspective.
• Score: 0.9176056742068812
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Polysemy and synonymy are two crucial interrelated facets of lexical ambiguity. While both phenomena have been studied extensively in NLP, leading to dedicated systems, they are often been considered independently. While many tasks dealing with polysemy (e.g. Word Sense Disambiguiation or Induction) highlight the role of a word's senses, the study of synonymy is rooted in the study of concepts, i.e. meaning shared across the lexicon. In this paper, we introduce Concept Induction, the unsupervised task of learning a soft clustering among words that defines a set of concepts directly from data. This task generalizes that of Word Sense Induction. We propose a bi-level approach to Concept Induction that leverages both a local lemma-centric view and a global cross-lexicon perspective to induce concepts. We evaluate the obtained clustering on SemCor's annotated data and obtain good performances (BCubed F1 above 0.60). We find that the local and the global levels are mutually beneficial to induce concepts and also senses in our setting. Finally, we create static embeddings representing our induced concepts and use them on the Word-in-Context task, obtaining competitive performances with the State-of-the-Art.

Related papers

• Identifying and interpreting non-aligned human conceptual representations using language modeling [0.0]
  We show that congenital blindness induces conceptual reorganization in both a-modal and sensory-related verbal domains. We find that blind individuals more strongly associate social and cognitive meanings to verbs related to motion. For some verbs, representations of blind and sighted are highly similar.
  arXiv  Detail & Related papers  (2024-03-10T13:02:27Z)
• Domain Embeddings for Generating Complex Descriptions of Concepts in Italian Language [65.268245109828]
  We propose a Distributional Semantic resource enriched with linguistic and lexical information extracted from electronic dictionaries. The resource comprises 21 domain-specific matrices, one comprehensive matrix, and a Graphical User Interface. Our model facilitates the generation of reasoned semantic descriptions of concepts by selecting matrices directly associated with concrete conceptual knowledge.
  arXiv  Detail & Related papers  (2024-02-26T15:04:35Z)
• Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation [100.81837601210597]
  We propose Concept Curation (CoCu) to bridge the gap between visual and textual semantics in pre-training data. CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin.
  arXiv  Detail & Related papers  (2023-09-24T00:05:39Z)
• Latent Topology Induction for Understanding Contextualized Representations [84.7918739062235]
  We study the representation space of contextualized embeddings and gain insight into the hidden topology of large language models. We show there exists a network of latent states that summarize linguistic properties of contextualized representations.
  arXiv  Detail & Related papers  (2022-06-03T11:22:48Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)
• Toward a Visual Concept Vocabulary for GAN Latent Space [74.12447538049537]
  This paper introduces a new method for building open-ended vocabularies of primitive visual concepts represented in a GAN's latent space. Our approach is built from three components: automatic identification of perceptually salient directions based on their layer selectivity; human annotation of these directions with free-form, compositional natural language descriptions. Experiments show that concepts learned with our approach are reliable and composable -- generalizing across classes, contexts, and observers.
  arXiv  Detail & Related papers  (2021-10-08T17:58:19Z)
• A Survey On Neural Word Embeddings [0.4822598110892847]
  The study of meaning in natural language processing relies on the distributional hypothesis. The revolutionary idea of distributed representation for a concept is close to the working of a human mind. Neural word embeddings transformed the whole field of NLP by introducing substantial improvements in all NLP tasks.
  arXiv  Detail & Related papers  (2021-10-05T03:37:57Z)
• Learning Zero-Shot Multifaceted Visually Grounded Word Embeddingsvia Multi-Task Training [8.271859911016719]
  Language grounding aims at linking the symbolic representation of language (e.g., words) into the rich perceptual knowledge of the outside world. We argue that this approach sacrifices the abstract knowledge obtained from linguistic co-occurrence statistics in the process of acquiring perceptual information.
  arXiv  Detail & Related papers  (2021-04-15T14:49:11Z)
• Joint Semantic Analysis with Document-Level Cross-Task Coherence Rewards [13.753240692520098]
  We present a neural network architecture for joint coreference resolution and semantic role labeling for English. We use reinforcement learning to encourage global coherence over the document and between semantic annotations. This leads to improvements on both tasks in multiple datasets from different domains.
  arXiv  Detail & Related papers  (2020-10-12T09:36:24Z)
• Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders [79.38278330678965]
  A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed. We propose a bi-encoder model that independently embeds (1) the target word with its surrounding context and (2) the dictionary definition, or gloss, of each sense.
  arXiv  Detail & Related papers  (2020-05-06T04:21:45Z)
• Decomposing Word Embedding with the Capsule Network [23.294890047230584]
  We propose a capsule network-based method to Decompose the unsupervised word Embedding of an ambiguous word into context specific Sense embedding. With attention operations, CapsDecE2S integrates the word context to reconstruct the multiple morpheme-like vectors into the context-specific sense embedding. In this method, we convert the sense learning into a binary classification that explicitly learns the relation between senses by the label of matching and non-matching.
  arXiv  Detail & Related papers  (2020-04-07T06:37:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.