From Topic to Transition Structure: Unsupervised Concept Discovery at Corpus Scale via Predictive Associative Memory

• URL: http://arxiv.org/abs/2603.18420v1
• Date: Thu, 19 Mar 2026 02:26:42 GMT
• Title: From Topic to Transition Structure: Unsupervised Concept Discovery at Corpus Scale via Predictive Associative Memory
• Abstract summary: We map pre-trained embeddings into an association space where passages with similar transition structure cluster together.<n>At k=100, clusters average 4,508 books each (of 9,766), confirming corpus-wide patterns.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embedding models group text by semantic content, what text is about. We show that temporal co-occurrence within texts discovers a different kind of structure: recurrent transition-structure concepts or what text does. We train a 29.4M-parameter contrastive model on 373 million co-occurrence pairs from 9,766 Project Gutenberg texts (24.96 million passages), mapping pre-trained embeddings into an association space where passages with similar transition structure cluster together. Under capacity constraint (42.75% accuracy), the model must compress across recurring patterns rather than memorise individual co-occurrences. Clustering at six granularities (k=50 to k=2,000) produces a multi-resolution concept map; from broad modes like "direct confrontation" and "lyrical meditation" to precise registers and scene templates like "sailor dialect" and "courtroom cross-examination." At k=100, clusters average 4,508 books each (of 9,766), confirming corpus-wide patterns. Direct comparison with embedding-similarity clustering shows that raw embeddings group by topic while association-space clusters group by function, register, and literary tradition. Unseen novels are assigned to existing clusters without retraining; the association model concentrates each novel into a selective subset of coherent clusters, while raw embedding assignment saturates nearly all clusters. Validation controls address positional, length, and book-concentration confounds. The method extends Predictive Associative Memory (PAM, arXiv:2602.11322) from episodic recall to concept formation: where PAM recalls specific associations, multi-epoch contrastive training under compression extracts structural patterns that transfer to unseen texts, the same framework producing qualitatively different behaviour in a different regime.

Related papers

• Mapping and Measuring the Behavioral Evolution of Large Language Models [28.603443859513153]
  Benchmark leaderboards summarize how well a language model performs, but not how its behavior relates to that of other models or changes across generations.<n>We characterize the output behavior of 32 models from six families using their responses to a shared bank of 10,000 prompts.
  arXiv  Detail & Related papers  (2026-08-11T15:07:28Z)
• Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training [12.81568269791334]
  We study whether related content is organized into coherent book-level documents.<n>We retrieve source material from a pre-training corpus, clusters it into topical units, plans hierarchical tables of contents, and assembles source-grounded sections into complete books.<n> Replacing natural books in a mid-training mix with this corpus improves downstream performance by +1.09 on average.
  arXiv  Detail & Related papers  (2026-07-30T12:15:41Z)
• Beyond Similarity: Grounded Agentic Extraction and Expert-Adjudicated Evaluation of Intertextuality in Classical Chinese Histories [3.694051864521789]
  We recast fine-grained intertextuality extraction as an agentic task in which a large language model reads two text units in full.<n>We validate the approach on an exhaustive comparison of the Analects with the Book of Han, where three domain experts adjudicate a pooled multi-model candidate set into a benchmark of 2,533 intertextual pairs.
  arXiv  Detail & Related papers  (2026-07-30T02:34:23Z)
• Self-Training Doesn't Flatten Language -- It Restructures It: Surface Markers Amplify While Deep Syntax Dies [4.738949927143789]
  Successive self-training on a language model's own outputs is widely characterized as a process of flattening.<n>We provide evidence that this characterization is incomplete.<n>Across eleven generations of self-training on five models, language is not flattened uniformly -- it is restructured.
  arXiv  Detail & Related papers  (2026-05-20T01:44:47Z)
• Cross-Layer Discrete Concept Discovery for Interpreting Language Models [13.842670153893977]
  Cross-layer VQ-VAE is a framework that uses vector quantization to map representations across layers.<n>Our approach uniquely combines top-k temperature-based sampling during quantization with EMA codebook updates.
  arXiv  Detail & Related papers  (2025-06-24T22:43:36Z)
• Self-supervised Latent Space Optimization with Nebula Variational Coding [87.20343320266215]
  This paper proposes a variational inference model which leads to a clustered embedding.<n>We introduce additional variables in the latent space, called textbfnebula anchors, that guide the latent variables to form clusters during training.<n>Since each latent feature can be labeled with the closest anchor, we also propose to apply metric learning in a self-supervised way to make the separation between clusters more explicit.
  arXiv  Detail & Related papers  (2025-06-02T08:13:32Z)
• Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management [0.0]
  FastText has established itself as a fundamental algorithm for learning word representations.<n>However, its hash-based bucketing mechanism introduces critical limitations for large-scale industrial deployment.<n>This paper presents a memory optimization framework that reimagines FastText's memory management.
  arXiv  Detail & Related papers  (2025-06-02T02:11:22Z)
• Hierarchical Level-Wise News Article Clustering via Multilingual Matryoshka Embeddings [5.161088104035108]
  We present a novel, scalable, interpretable, hierarchical, and multilingual approach to clustering news articles and social media data.<n>We first train multilingual Matryoshka embeddings that can determine story similarity at varying levels of granularity.<n>We develop an efficient hierarchical clustering algorithm that leverages the hierarchical nature of Matryoshka embeddings to identify unique news stories, narratives, and themes.
  arXiv  Detail & Related papers  (2025-05-30T22:17:18Z)
• Preserving Modality Structure Improves Multi-Modal Learning [64.10085674834252]
  Self-supervised learning on large-scale multi-modal datasets allows learning semantically meaningful embeddings without relying on human annotations. These methods often struggle to generalize well on out-of-domain data as they ignore the semantic structure present in modality-specific embeddings. We propose a novel Semantic-Structure-Preserving Consistency approach to improve generalizability by preserving the modality-specific relationships in the joint embedding space.
  arXiv  Detail & Related papers  (2023-08-24T20:46:48Z)
• Scaling Laws for Generative Mixed-Modal Language Models [103.25737824352949]
  We report new mixed-modal scaling laws that unify the contributions of individual modalities and the interactions between them. Specifically, we explicitly model the optimal synergy and competition due to data and model size as an additive term to previous uni-modal scaling laws. We also find four empirical phenomena observed during the training, such as emergent coordinate-ascent style training that naturally alternates between modalities.
  arXiv  Detail & Related papers  (2023-01-10T00:20:06Z)
• A Proposition-Level Clustering Approach for Multi-Document Summarization [82.4616498914049]
  We revisit the clustering approach, grouping together propositions for more precise information alignment. Our method detects salient propositions, clusters them into paraphrastic clusters, and generates a representative sentence for each cluster by fusing its propositions. Our summarization method improves over the previous state-of-the-art MDS method in the DUC 2004 and TAC 2011 datasets.
  arXiv  Detail & Related papers  (2021-12-16T10:34:22Z)
• Author Clustering and Topic Estimation for Short Texts [69.54017251622211]
  We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document. We also simultaneously cluster users, removing the need for post-hoc cluster estimation. Our method performs as well as -- or better -- than traditional approaches to problems arising in short text.
  arXiv  Detail & Related papers  (2021-06-15T20:55:55Z)
• Nutribullets Hybrid: Multi-document Health Summarization [36.95954983680022]
  We present a method for generating comparative summaries that highlights similarities and contradictions in input documents. Our framework leads to more faithful, relevant and aggregation-sensitive summarization -- while being equally fluent.
  arXiv  Detail & Related papers  (2021-04-08T01:44:29Z)
• A Framework for Authorial Clustering of Shorter Texts in Latent Semantic Spaces [4.18804572788063]
  Authorial clustering involves grouping documents written by the same author or team of authors without any prior positive examples of an author's writing style or thematic preferences. We propose a high-level framework which utilizes a compact data representation in a latent feature space derived with non-parametric topic modeling. We report on experiments with 120 collections in three languages and two genres and show that the topic-based latent feature space provides a promising level of performance.
  arXiv  Detail & Related papers  (2020-11-30T17:39:44Z)
• Relation Clustering in Narrative Knowledge Graphs [71.98234178455398]
  relational sentences in the original text are embedded (with SBERT) and clustered in order to merge together semantically similar relations. Preliminary tests show that such clustering might successfully detect similar relations, and provide a valuable preprocessing for semi-supervised approaches.
  arXiv  Detail & Related papers  (2020-11-27T10:43:04Z)
• LSD-C: Linearly Separable Deep Clusters [145.89790963544314]
  We present LSD-C, a novel method to identify clusters in an unlabeled dataset. Our method draws inspiration from recent semi-supervised learning practice and proposes to combine our clustering algorithm with self-supervised pretraining and strong data augmentation. We show that our approach significantly outperforms competitors on popular public image benchmarks including CIFAR 10/100, STL 10 and MNIST, as well as the document classification dataset Reuters 10K.
  arXiv  Detail & Related papers  (2020-06-17T17:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.