WOVe: Incorporating Word Order in GloVe Word Embeddings
- URL: http://arxiv.org/abs/2105.08597v1
- Date: Tue, 18 May 2021 15:28:20 GMT
- Title: WOVe: Incorporating Word Order in GloVe Word Embeddings
- Authors: Mohammed Ibrahim, Susan Gauch, Tyler Gerth, Brandon Cox
- Abstract summary: Defining a word as a vector makes it easy for machine learning algorithms to understand a text and extract information from it.
Word vector representations have been used in many applications such word synonyms, word analogy, syntactic parsing, and many others.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Word vector representations open up new opportunities to extract useful
information from unstructured text. Defining a word as a vector made it easy
for the machine learning algorithms to understand a text and extract
information from. Word vector representations have been used in many
applications such word synonyms, word analogy, syntactic parsing, and many
others. GloVe, based on word contexts and matrix vectorization, is an
ef-fective vector-learning algorithm. It improves on previous vector-learning
algorithms. However, the GloVe model fails to explicitly consider the order in
which words appear within their contexts. In this paper, multiple methods of
incorporating word order in GloVe word embeddings are proposed. Experimental
results show that our Word Order Vector (WOVe) word embeddings approach
outperforms unmodified GloVe on the natural lan-guage tasks of analogy
completion and word similarity. WOVe with direct concatenation slightly
outperformed GloVe on the word similarity task, increasing average rank by 2%.
However, it greatly improved on the GloVe baseline on a word analogy task,
achieving an average 36.34% improvement in accuracy.
Related papers
- Backpack Language Models [108.65930795825416]
We present Backpacks, a new neural architecture that marries strong modeling performance with an interface for interpretability and control.
We find that, after training, sense vectors specialize, each encoding a different aspect of a word.
We present simple algorithms that intervene on sense vectors to perform controllable text generation and debiasing.
arXiv Detail & Related papers (2023-05-26T09:26:23Z) - Tsetlin Machine Embedding: Representing Words Using Logical Expressions [10.825099126920028]
We introduce a Tsetlin Machine-based autoencoder that learns logical clauses self-supervised.
The clauses consist of contextual words like "black," "cup," and "hot" to define other words like "coffee"
We evaluate our embedding approach on several intrinsic and extrinsic benchmarks, outperforming GLoVe on six classification tasks.
arXiv Detail & Related papers (2023-01-02T15:02:45Z) - Simple, Interpretable and Stable Method for Detecting Words with Usage
Change across Corpora [54.757845511368814]
The problem of comparing two bodies of text and searching for words that differ in their usage arises often in digital humanities and computational social science.
This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large.
We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word.
arXiv Detail & Related papers (2021-12-28T23:46:00Z) - A Rule-based/BPSO Approach to Produce Low-dimensional Semantic Basis
Vectors Set [0.0]
In explicit semantic vectors, each dimension corresponds to a word, so word vectors are interpretable.
In this research, we propose a new approach to obtain low-dimensional explicit semantic vectors.
arXiv Detail & Related papers (2021-11-24T21:23:43Z) - Deriving Word Vectors from Contextualized Language Models using
Topic-Aware Mention Selection [46.97185212695267]
We propose a method for learning word representations that follows this basic strategy.
We take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts.
We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.
arXiv Detail & Related papers (2021-06-15T08:02:42Z) - SemGloVe: Semantic Co-occurrences for GloVe from BERT [55.420035541274444]
GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices.
We propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings.
arXiv Detail & Related papers (2020-12-30T15:38:26Z) - Accelerating Text Mining Using Domain-Specific Stop Word Lists [57.76576681191192]
We present a novel approach for the automatic extraction of domain-specific words called the hyperplane-based approach.
The hyperplane-based approach can significantly reduce text dimensionality by eliminating irrelevant features.
Results indicate that the hyperplane-based approach can reduce the dimensionality of the corpus by 90% and outperforms mutual information.
arXiv Detail & Related papers (2020-11-18T17:42:32Z) - Robust and Consistent Estimation of Word Embedding for Bangla Language
by fine-tuning Word2Vec Model [1.2691047660244335]
We analyze word2vec model for learning word vectors and present the most effective word embedding for Bangla language.
We cluster the word vectors to examine the relational similarity of words for intrinsic evaluation and also use different word embeddings as the feature of news article for extrinsic evaluation.
arXiv Detail & Related papers (2020-10-26T08:00:48Z) - Morphological Skip-Gram: Using morphological knowledge to improve word
representation [2.0129974477913457]
We propose a new method for training word embeddings by replacing the FastText bag of character n-grams for a bag of word morphemes.
The results show a competitive performance compared to FastText.
arXiv Detail & Related papers (2020-07-20T12:47:36Z) - Word Rotator's Distance [50.67809662270474]
Key principle in assessing textual similarity is measuring the degree of semantic overlap between two texts by considering the word alignment.
We show that the norm of word vectors is a good proxy for word importance, and their angle is a good proxy for word similarity.
We propose a method that first decouples word vectors into their norm and direction, and then computes alignment-based similarity.
arXiv Detail & Related papers (2020-04-30T17:48:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.