Identifying the Periodicity of Information in Natural Language

• URL: http://arxiv.org/abs/2510.27241v1
• Date: Fri, 31 Oct 2025 07:10:30 GMT
• Title: Identifying the Periodicity of Information in Natural Language
• Abstract summary: We introduce a new method called AutoPeriod of Surprisal (APS)<n>APS adopts a canonical periodicity detection algorithm and is able to identify any significant periods that exist in the surprisal sequence of a single document.<n>We conclude that the periodicity of information in language is a joint outcome from both structured factors and other driving factors that take effect at longer distances.
• Score: 6.222108497485838
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent theoretical advancement of information density in natural language has brought the following question on desk: To what degree does natural language exhibit periodicity pattern in its encoded information? We address this question by introducing a new method called AutoPeriod of Surprisal (APS). APS adopts a canonical periodicity detection algorithm and is able to identify any significant periods that exist in the surprisal sequence of a single document. By applying the algorithm to a set of corpora, we have obtained the following interesting results: Firstly, a considerable proportion of human language demonstrates a strong pattern of periodicity in information; Secondly, new periods that are outside the distributions of typical structural units in text (e.g., sentence boundaries, elementary discourse units, etc.) are found and further confirmed via harmonic regression modeling. We conclude that the periodicity of information in language is a joint outcome from both structured factors and other driving factors that take effect at longer distances. The advantages of our periodicity detection method and its potentials in LLM-generation detection are further discussed.

Related papers

• Repeated Sequences Reveal Gaps between Large Language Models and Natural Language [8.883733362171036]
  We show that GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size.<n>These results demonstrate that repeated-subsequence entropy provides a quantitative structural diagnostic.
  arXiv  Detail & Related papers  (2026-05-24T03:49:20Z)
• Priors in Time: Missing Inductive Biases for Language Model Interpretability [58.07412640266836]
  We show that Sparse Autoencoders impose priors that assume independence of concepts across time, implying stationarity.<n>We introduce a new interpretability objective -- Temporal Feature Analysis -- which possesses a temporal inductive bias to decompose representations at a given time into two parts.<n>Our results underscore the need for inductive biases that match the data in designing robust interpretability tools.
  arXiv  Detail & Related papers  (2025-11-03T18:43:48Z)
• Augmenting LLMs for General Time Series Understanding and Prediction [2.426309874608745]
  Time series data is fundamental to decision-making in many crucial domains including healthcare, finance, and environmental science.<n>We train this Time Series-augmented LLM (TsLLM) on a large corpus of over 2 million interleaved time series and text examples.<n>This training enables TsLLM to leverage both its language understanding and newly acquired temporal reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-01T16:54:46Z)
• The Harmonic Structure of Information Contours [54.38365999922221]
  We find consistent evidence of periodic patterns in information rate in texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese.<n>Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization.
  arXiv  Detail & Related papers  (2025-06-04T12:56:30Z)
• Linguistic Structure from a Bottleneck on Sequential Information Processing [5.850665541267672]
  We show that natural-language-like systematicity arises in codes that are constrained by predictive information. We show that human languages are structured to have low predictive information at the levels of phonology, morphology, syntax, and semantics.
  arXiv  Detail & Related papers  (2024-05-20T15:25:18Z)
• Complex systems approach to natural language [0.0]
  Review summarizes the main methodological concepts used in studying natural language from the perspective of complexity science. Three main complexity-related research trends in quantitative linguistics are covered.
  arXiv  Detail & Related papers  (2024-01-05T12:01:26Z)
• Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training [56.74440457571821]
  We analyze tasks covering syntax, semantics and reasoning, across 2M pre-training steps and five seeds. We identify critical learning phases across tasks and time, during which subspaces emerge, share information, and later disentangle to specialize. Our findings have implications for model interpretability, multi-task learning, and learning from limited data.
  arXiv  Detail & Related papers  (2023-10-25T09:09:55Z)
• Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models [18.874880342410876]
  We present Jamp, a Japanese benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments.
  arXiv  Detail & Related papers  (2023-06-19T07:00:14Z)
• O-Dang! The Ontology of Dangerous Speech Messages [53.15616413153125]
  We present O-Dang!: The Ontology of Dangerous Speech Messages, a systematic and interoperable Knowledge Graph (KG) O-Dang! is designed to gather and organize Italian datasets into a structured KG, according to the principles shared within the Linguistic Linked Open Data community. It provides a model for encoding both gold standard and single-annotator labels in the KG.
  arXiv  Detail & Related papers  (2022-07-13T11:50:05Z)
• On the probability-quality paradox in language generation [76.69397802617064]
  We analyze language generation through an information-theoretic lens. We posit that human-like language should contain an amount of information close to the entropy of the distribution over natural strings.
  arXiv  Detail & Related papers  (2022-03-31T17:43:53Z)
• Linguistic Typology Features from Text: Inferring the Sparse Features of World Atlas of Language Structures [73.06435180872293]
  We construct a recurrent neural network predictor based on byte embeddings and convolutional layers. We show that some features from various linguistic types can be predicted reliably.
  arXiv  Detail & Related papers  (2020-04-30T21:00:53Z)
• Information-Theoretic Probing for Linguistic Structure [74.04862204427944]
  We propose an information-theoretic operationalization of probing as estimating mutual information. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research.
  arXiv  Detail & Related papers  (2020-04-07T01:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.