A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings

• URL: http://arxiv.org/abs/2008.04702v1
• Date: Tue, 11 Aug 2020 13:54:11 GMT
• Title: A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings
• Authors: Lixing Zhu, Yulan He and Deyu Zhou
• Abstract summary: We propose a novel generative model to explore both local and global context for joint learning topics and topic-specific word embeddings. The trained model maps words to topic-dependent embeddings, which naturally addresses the issue of word polysemy.
• Score: 42.87769996249732
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a novel generative model to explore both local and global context for joint learning topics and topic-specific word embeddings. In particular, we assume that global latent topics are shared across documents, a word is generated by a hidden semantic vector encoding its contextual semantic meaning, and its context words are generated conditional on both the hidden semantic vector and global latent topics. Topics are trained jointly with the word embeddings. The trained model maps words to topic-dependent embeddings, which naturally addresses the issue of word polysemy. Experimental results show that the proposed model outperforms the word-level embedding methods in both word similarity evaluation and word sense disambiguation. Furthermore, the model also extracts more coherent topics compared with existing neural topic models or other models for joint learning of topics and word embeddings. Finally, the model can be easily integrated with existing deep contextualized word embedding learning methods to further improve the performance of downstream tasks such as sentiment classification.

Related papers

• CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling [23.323587005085564]
  We introduce a novel neural topic model called the Contextlized Word Topic Model (CWTM) CWTM integrates contextualized word embeddings from BERT. It is capable of learning the topic vector of a document without BOW information. It can also derive the topic vectors for individual words within a document based on their contextualized word embeddings.
  arXiv  Detail & Related papers  (2023-05-16T10:07:33Z)
• Topics in the Haystack: Extracting and Evaluating Topics beyond Coherence [0.0]
  We propose a method that incorporates a deeper understanding of both sentence and document themes. This allows our model to detect latent topics that may include uncommon words or neologisms. We present correlation coefficients with human identification of intruder words and achieve near-human level results at the word-intrusion task.
  arXiv  Detail & Related papers  (2023-03-30T12:24:25Z)
• Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding [143.5927158318524]
  Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. We introduce a new Compositional Temporal Grounding task and construct two new dataset splits. We argue that the inherent structured semantics inside the videos and language is the crucial factor to achieve compositional generalization.
  arXiv  Detail & Related papers  (2023-01-22T08:02:23Z)
• Topics in Contextualised Attention Embeddings [7.6650522284905565]
  Recent work has demonstrated that conducting clustering on the word-level contextual representations from a language model emulates word clusters that are discovered in latent topics of words from Latent Dirichlet Allocation. The important question is how such topical word clusters are automatically formed, through clustering, in the language model when it has not been explicitly designed to model latent topics. Using BERT and DistilBERT, we find that the attention framework plays a key role in modelling such word topic clusters.
  arXiv  Detail & Related papers  (2023-01-11T07:26:19Z)
• Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contexts [28.291684568220827]
  We propose an iterative framework, SeedTopicMine, which jointly learns from three types of contexts and fuses their context signals via an ensemble ranking process. Under various sets of seeds and on multiple datasets, SeedTopicMine consistently yields more coherent and accurate topics than existing seed-guided topic discovery approaches.
  arXiv  Detail & Related papers  (2022-12-12T16:03:38Z)
• HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding [54.52651110749165]
  We present a novel framework that introduces hyperbolic embeddings to represent words and topics. With the tree-likeness property of hyperbolic space, the underlying semantic hierarchy can be better exploited to mine more interpretable topics.
  arXiv  Detail & Related papers  (2022-10-16T02:54:17Z)
• Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation [59.01297461453444]
  We propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.
  arXiv  Detail & Related papers  (2022-05-26T13:26:03Z)
• Model Choices Influence Attributive Word Associations: A Semi-supervised Analysis of Static Word Embeddings [0.0]
  This work aims to assess attributive word associations across five different static word embedding architectures. Our results reveal that the choice of the context learning flavor during embedding training (CBOW vs skip-gram) impacts the word association distinguishability and word embeddings' sensitivity to deviations in the training corpora.
  arXiv  Detail & Related papers  (2020-12-14T22:27:18Z)
• A Survey on Contextual Embeddings [48.04732268018772]
  Contextual embeddings assign each word a representation based on its context, capturing uses of words across varied contexts and encoding knowledge that transfers across languages. We review existing contextual embedding models, cross-lingual polyglot pre-training, the application of contextual embeddings in downstream tasks, model compression, and model analyses.
  arXiv  Detail & Related papers  (2020-03-16T15:22:22Z)
• How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context [59.13515950353125]
  We present a grammar-based decoding semantic parsing and adapt typical context modeling methods on top of it. We evaluate 13 context modeling methods on two large cross-domain datasets, and our best model achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2020-02-03T11:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.