Embedding Words in Non-Vector Space with Unsupervised Graph Learning

• URL: http://arxiv.org/abs/2010.02598v1
• Date: Tue, 6 Oct 2020 10:17:49 GMT
• Title: Embedding Words in Non-Vector Space with Unsupervised Graph Learning
• Authors: Max Ryabinin, Sergei Popov, Liudmila Prokhorenkova, Elena Voita
• Abstract summary: We introduce GraphGlove: unsupervised graph word representations which are learned end-to-end. In our setting, each word is a node in a weighted graph and the distance between words is the shortest path distance between the corresponding nodes. We show that our graph-based representations substantially outperform vector-based methods on word similarity and analogy tasks.
• Score: 33.51809615505692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: It has become a de-facto standard to represent words as elements of a vector space (word2vec, GloVe). While this approach is convenient, it is unnatural for language: words form a graph with a latent hierarchical structure, and this structure has to be revealed and encoded by word embeddings. We introduce GraphGlove: unsupervised graph word representations which are learned end-to-end. In our setting, each word is a node in a weighted graph and the distance between words is the shortest path distance between the corresponding nodes. We adopt a recent method learning a representation of data in the form of a differentiable weighted graph and use it to modify the GloVe training algorithm. We show that our graph-based representations substantially outperform vector-based methods on word similarity and analogy tasks. Our analysis reveals that the structure of the learned graphs is hierarchical and similar to that of WordNet, the geometry is highly non-trivial and contains subgraphs with different local topology.

Related papers

• Graph Neural Networks on Discriminative Graphs of Words [19.817473565906777]
  In this work, we explore a new Discriminative Graph of Words Graph Neural Network (DGoW-GNN) approach to classify text. We propose a new model for the graph-based classification of text, which combines a GNN and a sequence model. We evaluate our approach on seven benchmark datasets and find that it is outperformed by several state-of-the-art baseline models.
  arXiv  Detail & Related papers  (2024-10-27T15:14:06Z)
• Conversational Semantic Parsing using Dynamic Context Graphs [68.72121830563906]
  We consider the task of conversational semantic parsing over general purpose knowledge graphs (KGs) with millions of entities, and thousands of relation-types. We focus on models which are capable of interactively mapping user utterances into executable logical forms.
  arXiv  Detail & Related papers  (2023-05-04T16:04:41Z)
• Semi-Supervised Hierarchical Graph Classification [54.25165160435073]
  We study the node classification problem in the hierarchical graph where a 'node' is a graph instance. We propose the Hierarchical Graph Mutual Information (HGMI) and present a way to compute HGMI with theoretical guarantee. We demonstrate the effectiveness of this hierarchical graph modeling and the proposed SEAL-CI method on text and social network data.
  arXiv  Detail & Related papers  (2022-06-11T04:05:29Z)
• CGMN: A Contrastive Graph Matching Network for Self-Supervised Graph Similarity Learning [65.1042892570989]
  We propose a contrastive graph matching network (CGMN) for self-supervised graph similarity learning. We employ two strategies, namely cross-view interaction and cross-graph interaction, for effective node representation learning. We transform node representations into graph-level representations via pooling operations for graph similarity computation.
  arXiv  Detail & Related papers  (2022-05-30T13:20:26Z)
• Hierarchical Heterogeneous Graph Representation Learning for Short Text Classification [60.233529926965836]
  We propose a new method called SHINE, which is based on graph neural network (GNN) for short text classification. First, we model the short text dataset as a hierarchical heterogeneous graph consisting of word-level component graphs. Then, we dynamically learn a short document graph that facilitates effective label propagation among similar short texts.
  arXiv  Detail & Related papers  (2021-10-30T05:33:05Z)
• Learning grounded word meaning representations on similarity graphs [16.422174125381762]
  This paper introduces a novel approach to learn visually grounded meaning representations of words. The lower level of the hierarchy models modality-specific word representations through dedicated but communicating graphs. The higher level puts these representations together on a single graph to learn a representation jointly from both modalities.
  arXiv  Detail & Related papers  (2021-09-07T13:40:32Z)
• Online Graph Dictionary Learning [10.394615068526505]
  We propose a new online Graph Dictionary Learning approach, which uses the Gromov Wasserstein divergence for the data fitting term. In our work, graphs are encoded through their nodes' pairwise relations and modeled as convex combination of graph atoms. Our approach naturally extends to labeled graphs, and is completed by a novel upper bound that can be used as a fast approximation of Gromov Wasserstein in the embedding space.
  arXiv  Detail & Related papers  (2021-02-12T14:39:28Z)
• COLOGNE: Coordinated Local Graph Neighborhood Sampling [1.6498361958317633]
  replacing discrete unordered objects such as graph nodes by real-valued vectors is at the heart of many approaches to learning from graph data. We address the problem of learning discrete node embeddings such that the coordinates of the node vector representations are graph nodes. This opens the door to designing interpretable machine learning algorithms for graphs as all attributes originally present in the nodes are preserved.
  arXiv  Detail & Related papers  (2021-02-09T11:39:06Z)
• Learning the Implicit Semantic Representation on Graph-Structured Data [57.670106959061634]
  Existing representation learning methods in graph convolutional networks are mainly designed by describing the neighborhood of each node as a perceptual whole. We propose a Semantic Graph Convolutional Networks (SGCN) that explores the implicit semantics by learning latent semantic-paths in graphs.
  arXiv  Detail & Related papers  (2021-01-16T16:18:43Z)
• Morphological Skip-Gram: Using morphological knowledge to improve word representation [2.0129974477913457]
  We propose a new method for training word embeddings by replacing the FastText bag of character n-grams for a bag of word morphemes. The results show a competitive performance compared to FastText.
  arXiv  Detail & Related papers  (2020-07-20T12:47:36Z)
• Graph-Structured Referring Expression Reasoning in The Wild [105.95488002374158]
  Grounding referring expressions aims to locate in an image an object referred to by a natural language expression. We propose a scene graph guided modular network (SGMN) to perform reasoning over a semantic graph and a scene graph. We also propose Ref-Reasoning, a large-scale real-world dataset for structured referring expression reasoning.
  arXiv  Detail & Related papers  (2020-04-19T11:00:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.