Word2Box: Learning Word Representation Using Box Embeddings

• URL: http://arxiv.org/abs/2106.14361v1
• Date: Mon, 28 Jun 2021 01:17:11 GMT
• Title: Word2Box: Learning Word Representation Using Box Embeddings
• Authors: Shib Sankar Dasgupta, Michael Boratko, Shriya Atmakuri, Xiang Lorraine Li, Dhruvesh Patel, Andrew McCallum
• Abstract summary: Learning vector representations for words is one of the most fundamental topics in NLP. Our model, Word2Box, takes a region-based approach to the problem of word representation, representing words as $n$-dimensional rectangles. We demonstrate improved performance on various word similarity tasks, particularly on less common words.
• Score: 28.080105878687185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning vector representations for words is one of the most fundamental topics in NLP, capable of capturing syntactic and semantic relationships useful in a variety of downstream NLP tasks. Vector representations can be limiting, however, in that typical scoring such as dot product similarity intertwines position and magnitude of the vector in space. Exciting innovations in the space of representation learning have proposed alternative fundamental representations, such as distributions, hyperbolic vectors, or regions. Our model, Word2Box, takes a region-based approach to the problem of word representation, representing words as $n$-dimensional rectangles. These representations encode position and breadth independently and provide additional geometric operations such as intersection and containment which allow them to model co-occurrence patterns vectors struggle with. We demonstrate improved performance on various word similarity tasks, particularly on less common words, and perform a qualitative analysis exploring the additional unique expressivity provided by Word2Box.

Related papers

• Representation Of Lexical Stylistic Features In Language Models' Embedding Space [28.60690854046176]
  We show that it is possible to derive a vector representation for each of these stylistic notions from only a small number of seed pairs. We conduct experiments on five datasets and find that static embeddings encode these features more accurately at the level of words and phrases. The lower performance of contextualized representations at the word level is partially attributable to the anisotropy of their vector space.
  arXiv  Detail & Related papers  (2023-05-29T23:44:26Z)
• Tsetlin Machine Embedding: Representing Words Using Logical Expressions [10.825099126920028]
  We introduce a Tsetlin Machine-based autoencoder that learns logical clauses self-supervised. The clauses consist of contextual words like "black," "cup," and "hot" to define other words like "coffee" We evaluate our embedding approach on several intrinsic and extrinsic benchmarks, outperforming GLoVe on six classification tasks.
  arXiv  Detail & Related papers  (2023-01-02T15:02:45Z)
• Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection [46.97185212695267]
  We propose a method for learning word representations that follows this basic strategy. We take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts. We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.
  arXiv  Detail & Related papers  (2021-06-15T08:02:42Z)
• Cross-Modal Discrete Representation Learning [73.68393416984618]
  We present a self-supervised learning framework that learns a representation that captures finer levels of granularity across different modalities. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities.
  arXiv  Detail & Related papers  (2021-06-10T00:23:33Z)
• The Low-Dimensional Linear Geometry of Contextualized Word Representations [27.50785941238007]
  We study the linear geometry of contextualized word representations in ELMO and BERT. We show that a variety of linguistic features are encoded in low-dimensional subspaces.
  arXiv  Detail & Related papers  (2021-05-15T00:58:08Z)
• High-dimensional distributed semantic spaces for utterances [0.2907403645801429]
  This paper describes a model for high-dimensional representation for utterance and text level data. It is based on a mathematically principled and behaviourally plausible approach to representing linguistic information. The paper shows how the implemented model is able to represent a broad range of linguistic features in a common integral framework of fixed dimensionality.
  arXiv  Detail & Related papers  (2021-04-01T12:09:47Z)
• Accurate Word Representations with Universal Visual Guidance [55.71425503859685]
  This paper proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. We build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
  arXiv  Detail & Related papers  (2020-12-30T09:11:50Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)
• Multidirectional Associative Optimization of Function-Specific Word Representations [86.87082468226387]
  We present a neural framework for learning associations between interrelated groups of words. Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO compositions lie close together. The model retains information about word group membership even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure.
  arXiv  Detail & Related papers  (2020-05-11T17:07:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.