Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling

• URL: http://arxiv.org/abs/2310.11772v3
• Date: Mon, 23 Oct 2023 06:05:59 GMT
• Title: Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling
• Abstract summary: We propose Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL) TSSP task is proposed to force the model to comprehend structural information by learning the original relations between adjacent sentences in a disarrayed document. CSSL objective is to ensure that the sentences representations in the same topic have higher similarity, while those in different topics are less similar.
• Score: 19.303419410130672
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Topic segmentation is critical for obtaining structured documents and improving downstream tasks such as information retrieval. Due to its ability of automatically exploring clues of topic shift from abundant labeled data, recent supervised neural models have greatly promoted the development of long document topic segmentation, but leaving the deeper relationship between coherence and topic segmentation underexplored. Therefore, this paper enhances the ability of supervised models to capture coherence from both logical structure and semantic similarity perspectives to further improve the topic segmentation performance, proposing Topic-aware Sentence Structure Prediction (TSSP) and Contrastive Semantic Similarity Learning (CSSL). Specifically, the TSSP task is proposed to force the model to comprehend structural information by learning the original relations between adjacent sentences in a disarrayed document, which is constructed by jointly disrupting the original document at topic and sentence levels. Moreover, we utilize inter- and intra-topic information to construct contrastive samples and design the CSSL objective to ensure that the sentences representations in the same topic have higher similarity, while those in different topics are less similar. Extensive experiments show that the Longformer with our approach significantly outperforms old state-of-the-art (SOTA) methods. Our approach improve $F_1$ of old SOTA by 3.42 (73.74 -> 77.16) and reduces $P_k$ by 1.11 points (15.0 -> 13.89) on WIKI-727K and achieves an average relative reduction of 4.3% on $P_k$ on WikiSection. The average relative $P_k$ drop of 8.38% on two out-of-domain datasets also demonstrates the robustness of our approach.

Related papers

• Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions [15.97570754056266]
  We propose a novel approach to construct semantically-grounded soft label targets using Language Models (LMs)<n>Our method produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus.<n>We also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents.
  arXiv  Detail & Related papers  (2026-02-20T00:12:04Z)
• Implementing Long Text Style Transfer with LLMs through Dual-Layered Sentence and Paragraph Structure Extraction and Mapping [6.445040420833822]
  We propose a hierarchical framework that combines sentence-level stylistic adaptation with paragraph-level structural coherence.<n>Our proposed framework, ZeroStylus, operates through two systematic phases: hierarchical template acquisition from reference texts and template-guided generation with multi-granular matching.
  arXiv  Detail & Related papers  (2025-05-11T05:53:33Z)
• Graph Topic Modeling for Documents with Spatial or Covariate Dependencies [0.9208007322096533]
  We address the challenge of incorporating document-level metadata into topic modeling. We propose a new estimator based on a fast graph-regularized iterative singular value decomposition. We validate our model through comprehensive experiments on synthetic datasets and three real-world corpora.
  arXiv  Detail & Related papers  (2024-12-19T03:00:26Z)
• Coherent Entity Disambiguation via Modeling Topic and Categorical Dependency [87.16283281290053]
  Previous entity disambiguation (ED) methods adopt a discriminative paradigm, where prediction is made based on matching scores between mention context and candidate entities. We propose CoherentED, an ED system equipped with novel designs aimed at enhancing the coherence of entity predictions. We achieve new state-of-the-art results on popular ED benchmarks, with an average improvement of 1.3 F1 points.
  arXiv  Detail & Related papers  (2023-11-06T16:40:13Z)
• Long Document Summarization with Top-down and Bottom-up Inference [113.29319668246407]
  We propose a principled inference framework to improve summarization models on two aspects. Our framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency. We demonstrate the effectiveness of the proposed framework on a diverse set of summarization datasets.
  arXiv  Detail & Related papers  (2022-03-15T01:24:51Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Author Clustering and Topic Estimation for Short Texts [69.54017251622211]
  We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document. We also simultaneously cluster users, removing the need for post-hoc cluster estimation. Our method performs as well as -- or better -- than traditional approaches to problems arising in short text.
  arXiv  Detail & Related papers  (2021-06-15T20:55:55Z)
• Topic Scaling: A Joint Document Scaling -- Topic Model Approach To Learn Time-Specific Topics [0.0]
  This paper proposes a new methodology to study sequential corpora by implementing a two-stage algorithm that learns time-based topics with respect to a scale of document positions. The first stage ranks documents using Wordfish to estimate document positions that serve as a dependent variable to learn relevant topics. The second stage ranks the inferred topics on the document scale to match their occurrences within the corpus and track their evolution.
  arXiv  Detail & Related papers  (2021-03-31T12:35:36Z)
• Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing [52.24507547010127]
  Cross-domain context-dependent semantic parsing is a new focus of research. We present a dynamic graph framework that effectively modelling contextual utterances, tokens, database schemas, and their complicated interaction as the conversation proceeds. The proposed framework outperforms all existing models by large margins, achieving new state-of-the-art performance on two large-scale benchmarks.
  arXiv  Detail & Related papers  (2021-01-05T18:11:29Z)
• Reasoning with Latent Structure Refinement for Document-Level Relation Extraction [20.308845516900426]
  We propose a novel model that empowers the relational reasoning across sentences by automatically inducing the latent document-level graph. Specifically, our model achieves an F1 score of 59.05 on a large-scale document-level dataset (DocRED)
  arXiv  Detail & Related papers  (2020-05-13T13:36:09Z)
• Text Segmentation by Cross Segment Attention [2.525236250247906]
  Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents. We establish a new state-of-the-art, reducing in particular the error rates by a large margin in all cases.
  arXiv  Detail & Related papers  (2020-04-30T01:36:52Z)
• A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining [52.11221075687124]
  We propose a novel abstractive summary network that adapts to the meeting scenario. We design a hierarchical structure to accommodate long meeting transcripts and a role vector to depict the difference among speakers. Our model outperforms previous approaches in both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-04-04T21:00:41Z)
• Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation [9.416757363901295]
  We introduce a novel supervised model for text segmentation with simple but explicit coherence modeling. Our model -- a neural architecture consisting of two hierarchically connected Transformer networks -- is a multi-task learning model that couples the sentence-level segmentation objective with the coherence objective that differentiates correct sequences of sentences from corrupt ones.
  arXiv  Detail & Related papers  (2020-01-03T17:06:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.