Tsetlin Machine Embedding: Representing Words Using Logical Expressions
- URL: http://arxiv.org/abs/2301.00709v1
- Date: Mon, 2 Jan 2023 15:02:45 GMT
- Title: Tsetlin Machine Embedding: Representing Words Using Logical Expressions
- Authors: Bimal Bhattarai and Ole-Christoffer Granmo and Lei Jiao and Rohan
Yadav and Jivitesh Sharma
- Abstract summary: We introduce a Tsetlin Machine-based autoencoder that learns logical clauses self-supervised.
The clauses consist of contextual words like "black," "cup," and "hot" to define other words like "coffee"
We evaluate our embedding approach on several intrinsic and extrinsic benchmarks, outperforming GLoVe on six classification tasks.
- Score: 10.825099126920028
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Embedding words in vector space is a fundamental first step in
state-of-the-art natural language processing (NLP). Typical NLP solutions
employ pre-defined vector representations to improve generalization by
co-locating similar words in vector space. For instance, Word2Vec is a
self-supervised predictive model that captures the context of words using a
neural network. Similarly, GLoVe is a popular unsupervised model incorporating
corpus-wide word co-occurrence statistics. Such word embedding has
significantly boosted important NLP tasks, including sentiment analysis,
document classification, and machine translation. However, the embeddings are
dense floating-point vectors, making them expensive to compute and difficult to
interpret. In this paper, we instead propose to represent the semantics of
words with a few defining words that are related using propositional logic. To
produce such logical embeddings, we introduce a Tsetlin Machine-based
autoencoder that learns logical clauses self-supervised. The clauses consist of
contextual words like "black," "cup," and "hot" to define other words like
"coffee," thus being human-understandable. We evaluate our embedding approach
on several intrinsic and extrinsic benchmarks, outperforming GLoVe on six
classification tasks. Furthermore, we investigate the interpretability of our
embedding using the logical representations acquired during training. We also
visualize word clusters in vector space, demonstrating how our logical
embedding co-locate similar words.
Related papers
- Searching for Discriminative Words in Multidimensional Continuous
Feature Space [0.0]
We propose a novel method to extract discriminative keywords from documents.
We show how different discriminative metrics influence the overall results.
We conclude that word feature vectors can substantially improve the topical inference of documents' meaning.
arXiv Detail & Related papers (2022-11-26T18:05:11Z) - Deriving Word Vectors from Contextualized Language Models using
Topic-Aware Mention Selection [46.97185212695267]
We propose a method for learning word representations that follows this basic strategy.
We take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts.
We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.
arXiv Detail & Related papers (2021-06-15T08:02:42Z) - WOVe: Incorporating Word Order in GloVe Word Embeddings [0.0]
Defining a word as a vector makes it easy for machine learning algorithms to understand a text and extract information from it.
Word vector representations have been used in many applications such word synonyms, word analogy, syntactic parsing, and many others.
arXiv Detail & Related papers (2021-05-18T15:28:20Z) - R$^2$-Net: Relation of Relation Learning Network for Sentence Semantic
Matching [58.72111690643359]
We propose a Relation of Relation Learning Network (R2-Net) for sentence semantic matching.
We first employ BERT to encode the input sentences from a global perspective.
Then a CNN-based encoder is designed to capture keywords and phrase information from a local perspective.
To fully leverage labels for better relation information extraction, we introduce a self-supervised relation of relation classification task.
arXiv Detail & Related papers (2020-12-16T13:11:30Z) - Modelling General Properties of Nouns by Selectively Averaging
Contextualised Embeddings [46.49372320363155]
We show how the contextualised embeddings predicted by BERT can be used to produce high-quality word vectors.
We find that a simple strategy of averaging the contextualised embeddings of masked word mentions leads to vectors that outperform the static word vectors.
arXiv Detail & Related papers (2020-12-04T14:03:03Z) - Robust and Consistent Estimation of Word Embedding for Bangla Language
by fine-tuning Word2Vec Model [1.2691047660244335]
We analyze word2vec model for learning word vectors and present the most effective word embedding for Bangla language.
We cluster the word vectors to examine the relational similarity of words for intrinsic evaluation and also use different word embeddings as the feature of news article for extrinsic evaluation.
arXiv Detail & Related papers (2020-10-26T08:00:48Z) - Using Holographically Compressed Embeddings in Question Answering [0.0]
This research employs holographic compression of pre-trained embeddings to represent a token, its part-of-speech, and named entity type.
The implementation, in a modified question answering recurrent deep learning network, shows that semantic relationships are preserved, and yields strong performance.
arXiv Detail & Related papers (2020-07-14T18:29:49Z) - Word Rotator's Distance [50.67809662270474]
Key principle in assessing textual similarity is measuring the degree of semantic overlap between two texts by considering the word alignment.
We show that the norm of word vectors is a good proxy for word importance, and their angle is a good proxy for word similarity.
We propose a method that first decouples word vectors into their norm and direction, and then computes alignment-based similarity.
arXiv Detail & Related papers (2020-04-30T17:48:42Z) - Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies [60.285091454321055]
We design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix.
On text classification, language modeling, and movie recommendation benchmarks, we show that ANT is particularly suitable for large vocabulary sizes.
arXiv Detail & Related papers (2020-03-18T13:07:51Z) - Lexical Sememe Prediction using Dictionary Definitions by Capturing
Local Semantic Correspondence [94.79912471702782]
Sememes, defined as the minimum semantic units of human languages, have been proven useful in many NLP tasks.
We propose a Sememe Correspondence Pooling (SCorP) model, which is able to capture this kind of matching to predict sememes.
We evaluate our model and baseline methods on a famous sememe KB HowNet and find that our model achieves state-of-the-art performance.
arXiv Detail & Related papers (2020-01-16T17:30:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.