Explainable and Discourse Topic-aware Neural Language Understanding

• URL: http://arxiv.org/abs/2006.10632v3
• Date: Tue, 27 Jun 2023 05:07:42 GMT
• Title: Explainable and Discourse Topic-aware Neural Language Understanding
• Authors: Yatin Chaudhary, Hinrich Sch\"utze, Pankaj Gupta
• Abstract summary: Marrying topic models and language models exposes language understanding to a broader source of document-level context beyond sentences. Existing approaches incorporate latent document topic proportions and ignore topical discourse in sentences of the document. We present a novel neural composite language model that exploits both the latent and explainable topics along with topical discourse at sentence-level.
• Score: 22.443597046878086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Marrying topic models and language models exposes language understanding to a broader source of document-level context beyond sentences via topics. While introducing topical semantics in language models, existing approaches incorporate latent document topic proportions and ignore topical discourse in sentences of the document. This work extends the line of research by additionally introducing an explainable topic representation in language understanding, obtained from a set of key terms correspondingly for each latent topic of the proportion. Moreover, we retain sentence-topic associations along with document-topic association by modeling topical discourse for every sentence in the document. We present a novel neural composite language model that exploits both the latent and explainable topics along with topical discourse at sentence-level in a joint learning framework of topic and language models. Experiments over a range of tasks such as language modeling, word sense disambiguation, document classification, retrieval and text generation demonstrate ability of the proposed model in improving language understanding.

Related papers

• Prompting Large Language Models for Topic Modeling [10.31712610860913]
  We propose PromptTopic, a novel topic modeling approach that harnesses the advanced language understanding of large language models (LLMs) It involves extracting topics at the sentence level from individual documents, then aggregating and condensing these topics into a predefined quantity, ultimately providing coherent topics for texts of varying lengths. We benchmark PromptTopic against the state-of-the-art baselines on three vastly diverse datasets, establishing its proficiency in discovering meaningful topics.
  arXiv  Detail & Related papers  (2023-12-15T11:15:05Z)
• Neural Conversation Models and How to Rein Them in: A Survey of Failures and Fixes [17.489075240435348]
  Recent conditional language models are able to continue any kind of text source in an often seemingly fluent way. From a linguistic perspective, contributing to a conversation is high. Recent approaches try to tame the underlying language models at various intervention points.
  arXiv  Detail & Related papers  (2023-08-11T12:07:45Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• Topics in the Haystack: Extracting and Evaluating Topics beyond Coherence [0.0]
  We propose a method that incorporates a deeper understanding of both sentence and document themes. This allows our model to detect latent topics that may include uncommon words or neologisms. We present correlation coefficients with human identification of intruder words and achieve near-human level results at the word-intrusion task.
  arXiv  Detail & Related papers  (2023-03-30T12:24:25Z)
• Topics in Contextualised Attention Embeddings [7.6650522284905565]
  Recent work has demonstrated that conducting clustering on the word-level contextual representations from a language model emulates word clusters that are discovered in latent topics of words from Latent Dirichlet Allocation. The important question is how such topical word clusters are automatically formed, through clustering, in the language model when it has not been explicitly designed to model latent topics. Using BERT and DistilBERT, we find that the attention framework plays a key role in modelling such word topic clusters.
  arXiv  Detail & Related papers  (2023-01-11T07:26:19Z)
• An Inclusive Notion of Text [69.36678873492373]
  We argue that clarity on the notion of text is crucial for reproducible and generalizable NLP. We introduce a two-tier taxonomy of linguistic and non-linguistic elements that are available in textual sources and can be used in NLP modeling.
  arXiv  Detail & Related papers  (2022-11-10T14:26:43Z)
• HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding [54.52651110749165]
  We present a novel framework that introduces hyperbolic embeddings to represent words and topics. With the tree-likeness property of hyperbolic space, the underlying semantic hierarchy can be better exploited to mine more interpretable topics.
  arXiv  Detail & Related papers  (2022-10-16T02:54:17Z)
• Knowledge-Aware Bayesian Deep Topic Model [50.58975785318575]
  We propose a Bayesian generative model for incorporating prior domain knowledge into hierarchical topic modeling. Our proposed model efficiently integrates the prior knowledge and improves both hierarchical topic discovery and document representation.
  arXiv  Detail & Related papers  (2022-09-20T09:16:05Z)
• Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning [50.08729005865331]
  This paper develops a plug-and-play hierarchical-topic-guided image paragraph generation framework. To capture the correlations between the image and text at multiple levels of abstraction, we design a variational inference network. To guide the paragraph generation, the learned hierarchical topics and visual features are integrated into the language model.
  arXiv  Detail & Related papers  (2021-05-10T06:55:39Z)
• Topic-Aware Multi-turn Dialogue Modeling [91.52820664879432]
  This paper presents a novel solution for multi-turn dialogue modeling, which segments and extracts topic-aware utterances in an unsupervised way. Our topic-aware modeling is implemented by a newly proposed unsupervised topic-aware segmentation algorithm and Topic-Aware Dual-attention Matching (TADAM) Network.
  arXiv  Detail & Related papers  (2020-09-26T08:43:06Z)
• A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings [42.87769996249732]
  We propose a novel generative model to explore both local and global context for joint learning topics and topic-specific word embeddings. The trained model maps words to topic-dependent embeddings, which naturally addresses the issue of word polysemy.
  arXiv  Detail & Related papers  (2020-08-11T13:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.