Hierarchical Graph Topic Modeling with Topic Tree-based Transformer

• URL: http://arxiv.org/abs/2502.11345v1
• Date: Mon, 17 Feb 2025 01:55:29 GMT
• Title: Hierarchical Graph Topic Modeling with Topic Tree-based Transformer
• Authors: Delvin Ce Zhang, Menglin Yang, Xiaobao Wu, Jiasheng Zhang, Hady W. Lauw,
• Abstract summary: We propose a Hierarchical Graph Topic Modeling Transformer to integrate both topic hierarchy within documents and graph hierarchy across documents. To preserve both topic and graph hierarchies, we design our model in hyperbolic space and propose Hyperbolic Doubly Recurrent Neural Network. Both supervised and unsupervised experiments verify the effectiveness of our model.
• Score: 35.554979581137296
• License:
• Abstract: Textual documents are commonly connected in a hierarchical graph structure where a central document links to others with an exponentially growing connectivity. Though Hyperbolic Graph Neural Networks (HGNNs) excel at capturing such graph hierarchy, they cannot model the rich textual semantics within documents. Moreover, text contents in documents usually discuss topics of different specificity. Hierarchical Topic Models (HTMs) discover such latent topic hierarchy within text corpora. However, most of them focus on the textual content within documents, and ignore the graph adjacency across interlinked documents. We thus propose a Hierarchical Graph Topic Modeling Transformer to integrate both topic hierarchy within documents and graph hierarchy across documents into a unified Transformer. Specifically, to incorporate topic hierarchy within documents, we design a topic tree and infer a hierarchical tree embedding for hierarchical topic modeling. To preserve both topic and graph hierarchies, we design our model in hyperbolic space and propose Hyperbolic Doubly Recurrent Neural Network, which models ancestral and fraternal tree structure. Both hierarchies are inserted into each Transformer layer to learn unified representations. Both supervised and unsupervised experiments verify the effectiveness of our model.

Related papers

• Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classification [20.434941308959786]
  Long document classification presents challenges due to their extensive content and complex structure. Existing methods often struggle with token limits and fail to adequately model hierarchical relationships within documents. Our approach integrates syntax trees for sentence encodings and document graphs for document encodings, which capture fine-grained syntactic relationships and broader document contexts.
  arXiv  Detail & Related papers  (2024-10-03T19:25:01Z)
• HDT: Hierarchical Document Transformer [70.2271469410557]
  HDT exploits document structure by introducing auxiliary anchor tokens and redesigning the attention mechanism into a sparse multi-level hierarchy. We develop a novel sparse attention kernel that considers the hierarchical structure of documents.
  arXiv  Detail & Related papers  (2024-07-11T09:28:04Z)
• Modeling Unified Semantic Discourse Structure for High-quality Headline Generation [45.23071138765902]
  We propose using a unified semantic discourse structure (S3) to represent document semantics. The hierarchical composition of sentence, clause, and word intrinsically characterizes the semantic meaning of the overall document. Our work can be instructive for a broad range of document modeling tasks, more than headline or summarization generation.
  arXiv  Detail & Related papers  (2024-03-23T09:18:53Z)
• On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling [29.277151061615434]
  We propose Transport Plan and Context-aware Hierarchical Topic Model (TraCo) TraCo constrains dependencies to ensure their sparsity and balance, and also regularizes topic hierarchy building with them. Rather than previously entangled decoding, it distributes different semantic granularity to topics at different levels by disentangled decoding.
  arXiv  Detail & Related papers  (2024-01-25T11:47:58Z)
• HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding [54.52651110749165]
  We present a novel framework that introduces hyperbolic embeddings to represent words and topics. With the tree-likeness property of hyperbolic space, the underlying semantic hierarchy can be better exploited to mine more interpretable topics.
  arXiv  Detail & Related papers  (2022-10-16T02:54:17Z)
• Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones [64.75766944882389]
  We present ConE (Cone Embedding), a KG embedding model that is able to simultaneously model multiple hierarchical as well as non-hierarchical relations in a knowledge graph. In particular, ConE uses cone containment constraints in different subspaces of the hyperbolic embedding space to capture multiple heterogeneous hierarchies. Our approach yields new state-of-the-art Hits@1 of 45.3% on WN18RR and 16.1% on DDB14 (0.231 MRR)
  arXiv  Detail & Related papers  (2021-10-28T07:16:08Z)
• TopicNet: Semantic Graph-Guided Topic Discovery [51.71374479354178]
  Existing deep hierarchical topic models are able to extract semantically meaningful topics from a text corpus in an unsupervised manner. We introduce TopicNet as a deep hierarchical topic model that can inject prior structural knowledge as an inductive bias to influence learning.
  arXiv  Detail & Related papers  (2021-10-27T09:07:14Z)
• Hierarchical Topic Mining via Joint Spherical Tree and Text Embedding [37.7780399311715]
  Hierarchical Topic Mining aims to mine a set of representative terms for each category from a text corpus to help a user comprehend his/her interested topics. Our model, named JoSH, mines a high-quality set of hierarchical topics with high efficiency and benefits weakly-supervised hierarchical text classification tasks.
  arXiv  Detail & Related papers  (2020-07-18T23:30:47Z)
• Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning [72.52804406378023]
  Cross-modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. To improve fine-grained video-text retrieval, we propose a Hierarchical Graph Reasoning model, which decomposes video-text matching into global-to-local levels.
  arXiv  Detail & Related papers  (2020-03-01T03:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.