The Harmonic Structure of Information Contours

• URL: http://arxiv.org/abs/2506.03902v1
• Date: Wed, 04 Jun 2025 12:56:30 GMT
• Title: The Harmonic Structure of Information Contours
• Abstract summary: We find consistent evidence of periodic patterns in information rate in texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese.<n>Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization.
• Score: 54.38365999922221
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The uniform information density (UID) hypothesis proposes that speakers aim to distribute information evenly throughout a text, balancing production effort and listener comprehension difficulty. However, language typically does not maintain a strictly uniform information rate; instead, it fluctuates around a global average. These fluctuations are often explained by factors such as syntactic constraints, stylistic choices, or audience design. In this work, we explore an alternative perspective: that these fluctuations may be influenced by an implicit linguistic pressure towards periodicity, where the information rate oscillates at regular intervals, potentially across multiple frequencies simultaneously. We apply harmonic regression and introduce a novel extension called time scaling to detect and test for such periodicity in information contours. Analyzing texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese, we find consistent evidence of periodic patterns in information rate. Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization. Beyond highlighting the connection between information rate and discourse structure, our approach offers a general framework for uncovering structural pressures at various levels of linguistic granularity.

Related papers

• Evolving language compositionality in a frequency-structured meaning space [1.5749416770494706]
  We show how changing the frequency of different meanings affects the character of its compositionality.<n>Results suggest that frequency can shape emergent linguistic structure only when the frequency distribution is defined over form-meaning units.
  arXiv  Detail & Related papers  (2026-07-31T17:18:57Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization [19.24080183297062]
  Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies.<n>We study what determines this frequency usage and propose a data-centered explanation.
  arXiv  Detail & Related papers  (2026-07-08T17:38:14Z)
• Causally Evaluating the Learnability of Formal Language Tasks [59.73001113068827]
  We rigorously investigate the relationship between data frequency and learnability.<n>We derive Kullback-Leibler divergence metrics to measure the learnability of specific sub-tasks.<n>Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions.
  arXiv  Detail & Related papers  (2026-06-08T17:58:36Z)
• Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system [1.4363133217553712]
  We quantify scale-dependent fluctuations along the token sequence using an embedding-step signal.<n>Across multiple languages and corpora, the resulting power spectrum exhibits a robust power law with an exponent close to $5/3$.<n>By analogy with the Kolmogorov spectrum in turbulence, our findings suggest that semantic information is integrated in a scale-free, self-similar manner across linguistic scales.
  arXiv  Detail & Related papers  (2026-04-07T07:36:12Z)
• Is Information Density Uniform when Utterances are Grounded on Perception and Discourse? [7.3258783042969675]
  We present the first computational study of Uniform Information Density (UID) in visually grounded settings.<n>We find that grounding on perception consistently smooths the distribution of information, increasing both global and local uniformity.<n>Overall, this study takes a first step towards modelling the temporal dynamics of information flow in ecologically plausible, multimodal language use.
  arXiv  Detail & Related papers  (2026-02-16T11:25:00Z)
• Identifying the Periodicity of Information in Natural Language [6.222108497485838]
  We introduce a new method called AutoPeriod of Surprisal (APS)<n>APS adopts a canonical periodicity detection algorithm and is able to identify any significant periods that exist in the surprisal sequence of a single document.<n>We conclude that the periodicity of information in language is a joint outcome from both structured factors and other driving factors that take effect at longer distances.
  arXiv  Detail & Related papers  (2025-10-31T07:10:30Z)
• Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics [56.145578792496714]
  Large language models (LLMs) struggle with cross-lingual knowledge transfer.<n>We study the causes and dynamics of this phenomenon by training small Transformer models from scratch on synthetic multilingual datasets.
  arXiv  Detail & Related papers  (2025-08-14T18:44:13Z)
• Towards Explainable Bilingual Multimodal Misinformation Detection and Localization [64.37162720126194]
  BiMi is a framework that jointly performs region-level localization, cross-modal and cross-lingual consistency detection, and natural language explanation for misinformation analysis.<n>BiMiBench is a benchmark constructed by systematically editing real news images and subtitles.<n>BiMi outperforms strong baselines by up to +8.9 in classification accuracy, +15.9 in localization accuracy, and +2.5 in explanation BERTScore.
  arXiv  Detail & Related papers  (2025-06-28T15:43:06Z)
• Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent [22.63155507847401]
  We predict that languages that use prosody to make lexical distinctions should exhibit a higher mutual information between word identity and prosody, compared to languages that don't.<n>We use a dataset of speakers reading sentences aloud in ten languages across five language families to estimate the mutual information between the text and their pitch curves.
  arXiv  Detail & Related papers  (2025-05-12T15:25:17Z)
• Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias [76.85949078144098]
  This paper focuses on textual hallucinations, where diffusion models correctly generate individual symbols but assemble them in a nonsensical manner.<n>We observe that such phenomenon is attributed it to the network's local generation bias.<n>We also theoretically analyze the training dynamics for a specific case involving a two-layer learning parity points on a hypercube.
  arXiv  Detail & Related papers  (2025-03-05T15:28:50Z)
• Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts [4.796752450839119]
  We develop a framework to generalize classifiers over time on multiple languages.<n>Our analysis shows classification performance varies over time across different languages.<n>Our study provides analytic insights and addresses the need for time-aware models.
  arXiv  Detail & Related papers  (2025-02-12T22:30:18Z)
• Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse [54.08750245737734]
  We propose that speakers modulate information rate based on location within a hierarchically-structured model of discourse. We find that hierarchical predictors are significant predictors of a discourse's information contour and that deeply nested hierarchical predictors are more predictive than shallow ones.
  arXiv  Detail & Related papers  (2024-10-21T14:42:37Z)
• On the Role of Context in Reading Time Prediction [50.87306355705826]
  We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit is an affine function of its in-context information content.
  arXiv  Detail & Related papers  (2024-09-12T15:52:22Z)
• Decoding Multilingual Topic Dynamics and Trend Identification through ARIMA Time Series Analysis on Social Networks: A Novel Data Translation Framework Enhanced by LDA/HDP Models [0.08246494848934444]
  We focus on dialogues within Tunisian social networks during the Coronavirus Pandemic and other notable themes like sports and politics. We start by aggregating a varied multilingual corpus of comments relevant to these subjects. We then introduce our No-English-to-English Machine Translation approach to handle linguistic differences.
  arXiv  Detail & Related papers  (2024-03-18T00:01:10Z)
• Putting Context in Context: the Impact of Discussion Structure on Text Classification [13.15873889847739]
  We propose a series of experiments on a large dataset for stance detection in English. We evaluate the contribution of different types of contextual information. We show that structural information can be highly beneficial to text classification but only under certain circumstances.
  arXiv  Detail & Related papers  (2024-02-05T12:56:22Z)
• How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study [59.13867562744973]
  This work systematically assesses LMs' capabilities for out-of-distribution (OOD) scenarios. We find that the efficacy of such learning paradigms varies with the type of OOD. Specifically, while ICL excels for domain shifts, prompt-based fine-tuning surpasses for topic shifts.
  arXiv  Detail & Related papers  (2023-09-15T11:15:47Z)
• Generic Temporal Reasoning with Differential Analysis and Explanation [61.96034987217583]
  We introduce a novel task named TODAY that bridges the gap with temporal differential analysis. TODAY evaluates whether systems can correctly understand the effect of incremental changes. We show that TODAY's supervision style and explanation annotations can be used in joint learning.
  arXiv  Detail & Related papers  (2022-12-20T17:40:03Z)
• Locally Typical Sampling [84.62530743899025]
  We show that today's probabilistic language generators fall short when it comes to producing coherent and fluent text.<n>We propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models.
  arXiv  Detail & Related papers  (2022-02-01T18:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.