Principal Word Vectors
- URL: http://arxiv.org/abs/2007.04629v1
- Date: Thu, 9 Jul 2020 08:29:57 GMT
- Title: Principal Word Vectors
- Authors: Ali Basirat, Christian Hardmeier, Joakim Nivre
- Abstract summary: We generalize principal component analysis for embedding words into a vector space.
We show that the spread and the discriminability of the principal word vectors are higher than that of other word embedding methods.
- Score: 5.64434321651888
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We generalize principal component analysis for embedding words into a vector
space. The generalization is made in two major levels. The first is to
generalize the concept of the corpus as a counting process which is defined by
three key elements vocabulary set, feature (annotation) set, and context. This
generalization enables the principal word embedding method to generate word
vectors with regard to different types of contexts and different types of
annotations provided for a corpus. The second is to generalize the
transformation step used in most of the word embedding methods. To this end, we
define two levels of transformations. The first is a quadratic transformation,
which accounts for different types of weighting over the vocabulary units and
contextual features. Second is an adaptive non-linear transformation, which
reshapes the data distribution to be meaningful to principal component
analysis. The effect of these generalizations on the word vectors is
intrinsically studied with regard to the spread and the discriminability of the
word vectors. We also provide an extrinsic evaluation of the contribution of
the principal word vectors on a word similarity benchmark and the task of
dependency parsing. Our experiments are finalized by a comparison between the
principal word vectors and other sets of word vectors generated with popular
word embedding methods. The results obtained from our intrinsic evaluation
metrics show that the spread and the discriminability of the principal word
vectors are higher than that of other word embedding methods. The results
obtained from the extrinsic evaluation metrics show that the principal word
vectors are better than some of the word embedding methods and on par with
popular methods of word embedding.
Related papers
- Backpack Language Models [108.65930795825416]
We present Backpacks, a new neural architecture that marries strong modeling performance with an interface for interpretability and control.
We find that, after training, sense vectors specialize, each encoding a different aspect of a word.
We present simple algorithms that intervene on sense vectors to perform controllable text generation and debiasing.
arXiv Detail & Related papers (2023-05-26T09:26:23Z) - Contextualized Word Vector-based Methods for Discovering Semantic
Differences with No Training nor Word Alignment [17.229611956178818]
We propose methods for discovering semantic differences in words appearing in two corpora.
The key idea is that the coverage of meanings is reflected in the norm of its mean word vector.
We show these advantages for native and non-native English corpora and also for historical corpora.
arXiv Detail & Related papers (2023-05-19T08:27:17Z) - Describing Sets of Images with Textual-PCA [89.46499914148993]
We seek to semantically describe a set of images, capturing both the attributes of single images and the variations within the set.
Our procedure is analogous to Principle Component Analysis, in which the role of projection vectors is replaced with generated phrases.
arXiv Detail & Related papers (2022-10-21T17:10:49Z) - Simple, Interpretable and Stable Method for Detecting Words with Usage
Change across Corpora [54.757845511368814]
The problem of comparing two bodies of text and searching for words that differ in their usage arises often in digital humanities and computational social science.
This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large.
We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word.
arXiv Detail & Related papers (2021-12-28T23:46:00Z) - Deriving Word Vectors from Contextualized Language Models using
Topic-Aware Mention Selection [46.97185212695267]
We propose a method for learning word representations that follows this basic strategy.
We take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts.
We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.
arXiv Detail & Related papers (2021-06-15T08:02:42Z) - WOVe: Incorporating Word Order in GloVe Word Embeddings [0.0]
Defining a word as a vector makes it easy for machine learning algorithms to understand a text and extract information from it.
Word vector representations have been used in many applications such word synonyms, word analogy, syntactic parsing, and many others.
arXiv Detail & Related papers (2021-05-18T15:28:20Z) - Robust and Consistent Estimation of Word Embedding for Bangla Language
by fine-tuning Word2Vec Model [1.2691047660244335]
We analyze word2vec model for learning word vectors and present the most effective word embedding for Bangla language.
We cluster the word vectors to examine the relational similarity of words for intrinsic evaluation and also use different word embeddings as the feature of news article for extrinsic evaluation.
arXiv Detail & Related papers (2020-10-26T08:00:48Z) - Unsupervised Distillation of Syntactic Information from Contextualized
Word Representations [62.230491683411536]
We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations.
To this end, we automatically generate groups of sentences which are structurally similar but semantically different.
We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
arXiv Detail & Related papers (2020-10-11T15:13:18Z) - Word Rotator's Distance [50.67809662270474]
Key principle in assessing textual similarity is measuring the degree of semantic overlap between two texts by considering the word alignment.
We show that the norm of word vectors is a good proxy for word importance, and their angle is a good proxy for word similarity.
We propose a method that first decouples word vectors into their norm and direction, and then computes alignment-based similarity.
arXiv Detail & Related papers (2020-04-30T17:48:42Z) - Generating Word and Document Embeddings for Sentiment Analysis [0.36525095710982913]
In this paper, we combine contextual and supervised information with the general semantic representations of words occurring in the dictionary.
We induce domain-specific sentimental vectors for two corpora, which are the movie domain and the Twitter datasets in Turkish.
It shows that our approaches are cross-domain and portable to other languages.
arXiv Detail & Related papers (2020-01-05T16:34:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.