Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasing

• URL: http://arxiv.org/abs/2502.15208v1
• Date: Fri, 21 Feb 2025 04:46:57 GMT
• Title: Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasing
• Abstract summary: Repetitive transformations can lead to stable configurations, known as attractors, including fixed points and limit cycles.<n>Applying this perspective to large language models (LLMs), which iteratively map input text to output text, provides a principled approach to characterizing long-term behaviors.<n>Successive paraphrasing serves as a compelling testbed for exploring such dynamics, as paraphrases re-express the same underlying meaning with linguistic variation.
• Score: 28.646627695015646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dynamical systems theory provides a framework for analyzing iterative processes and evolution over time. Within such systems, repetitive transformations can lead to stable configurations, known as attractors, including fixed points and limit cycles. Applying this perspective to large language models (LLMs), which iteratively map input text to output text, provides a principled approach to characterizing long-term behaviors. Successive paraphrasing serves as a compelling testbed for exploring such dynamics, as paraphrases re-express the same underlying meaning with linguistic variation. Although LLMs are expected to explore a diverse set of paraphrases in the text space, our study reveals that successive paraphrasing converges to stable periodic states, such as 2-period attractor cycles, limiting linguistic diversity. This phenomenon is attributed to the self-reinforcing nature of LLMs, as they iteratively favour and amplify certain textual forms over others. This pattern persists with increasing generation randomness or alternating prompts and LLMs. These findings underscore inherent constraints in LLM generative capability, while offering a novel dynamical systems perspective for studying their expressive potential.

Related papers

• SWIM: Student Writing Simulation via Proficiency-Conditioned Generation [54.69949380133243]
  This work explores if language models can realistically simulate student writing, and introduce SWIM.<n>We evaluate prompting, supervised fine-tuning, and reinforcement learning methods for writing simulation using automated essay scoring as a measure of profile alignment.<n>Our findings suggest that explicit supervision enables substantially stronger profile alignment than prompting alone, while authentic low-proficiency writing remains challenging to reproduce.
  arXiv  Detail & Related papers  (2026-09-02T23:10:51Z)
• Repeated Sequences Reveal Gaps between Large Language Models and Natural Language [8.883733362171036]
  We show that GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size.<n>These results demonstrate that repeated-subsequence entropy provides a quantitative structural diagnostic.
  arXiv  Detail & Related papers  (2026-05-24T03:49:20Z)
• Markovian Generation Chains in Large Language Models [26.873145186799608]
  We study how texts evolve when they are repeatedly processed by large language models (LLMs)<n>In this paper, we define this iterative inference process as Markovian generation chains.<n>We show that iterative process can either increase or reduce sentence diversity depending on factors such as the temperature parameter and the initial input sentence.
  arXiv  Detail & Related papers  (2026-03-11T18:54:17Z)
• Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs [100.02824137397464]
  We investigate how Large Language Models adapt their internal representations when encountering inputs of increasing difficulty.<n>We reveal a consistent and quantifiable phenomenon: as task difficulty increases, the last hidden states of LLMs become substantially sparser.<n>This sparsity--difficulty relation is observable across diverse models and domains.
  arXiv  Detail & Related papers  (2026-03-03T18:48:15Z)
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs [49.66344956133349]
  Reasoning capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models.<n>This paper proposes Reasoning Palette, a novel latent-modulation framework that endows the model with a latent variable for strategic contextualization.
  arXiv  Detail & Related papers  (2025-12-19T03:32:53Z)
• STELLA: Guiding Large Language Models for Time Series Forecasting with Semantic Abstractions [4.169671705130711]
  We propose STELLA, a framework that mines and injects structured supplementary and complementary information.<n>StELLA employs a dynamic semantic abstraction mechanism that decouples input series into trend, seasonality, and residual components.<n>Experiments on eight benchmark datasets demonstrate that STELLA outperforms state-of-the-art methods in long- and short-term forecasting.
  arXiv  Detail & Related papers  (2025-12-04T14:56:36Z)
• Learning Evolving Latent Strategies for Multi-Agent Language Systems without Model Fine-Tuning [0.6345523830122167]
  This study proposes a multi-agent language framework that enables continual strategy evolution without fine-tuning the language model's parameters.<n>We construct a dual-loop architecture: the behavior loop adjusts action preferences based on environmental rewards, while the language loop updates the external latent vectors by reflecting on the semantic embeddings of generated text.
  arXiv  Detail & Related papers  (2025-11-28T23:36:45Z)
• How Do LLM-Generated Texts Impact Term-Based Retrieval Models? [76.92519309816008]
  This paper investigates the influence of large language models (LLMs) on term-based retrieval models.<n>Our linguistic analysis reveals that LLM-generated texts exhibit smoother high-frequency and steeper low-frequency Zipf slopes.<n>Our study further explores whether term-based retrieval models demonstrate source bias, concluding that these models prioritize documents whose term distributions closely correspond to those of the queries.
  arXiv  Detail & Related papers  (2025-08-25T06:43:27Z)
• Semantic-Enhanced Time-Series Forecasting via Large Language Models [20.383296465541758]
  Time series forecasting plays a significant role in finance, energy, meteorology, and IoT applications.<n>Recent studies have leveraged the generalization capabilities of large language models (LLMs) to adapt to time series forecasting, achieving promising performance.<n>We propose a novel Semantic-Enhanced LLM (SE-LLM) that explores the inherent periodicity and anomalous characteristics of time series to embed into the semantic space.
  arXiv  Detail & Related papers  (2025-08-11T07:19:21Z)
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey [69.45421620616486]
  This work presents the first structured taxonomy and analysis of discrete tokenization methods designed for large language models (LLMs)<n>We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines.<n>We identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints.
  arXiv  Detail & Related papers  (2025-07-21T10:52:14Z)
• Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model [26.655013761142758]
  Current Multimodal Large Language Models (MLLMs) struggle with periodic tasks due to limitations in: 1) lack of temporal modelling and 2) conflict between short and long periods.<n>This paper introduces Period-LLM, a multimodal large language model designed to enhance the performance of periodic tasks across various modalities.
  arXiv  Detail & Related papers  (2025-05-30T11:23:21Z)
• LLM-PS: Empowering Large Language Models for Time Series Forecasting with Temporal Patterns and Semantics [56.99021951927683]
  Time Series Forecasting (TSF) is critical in many real-world domains like financial planning and health monitoring. Existing Large Language Models (LLMs) usually perform suboptimally because they neglect the inherent characteristics of time series data. We propose LLM-PS to empower the LLM for TSF by learning the fundamental textitPatterns and meaningful textitSemantics from time series data.
  arXiv  Detail & Related papers  (2025-03-12T11:45:11Z)
• LogiDynamics: Unraveling the Dynamics of Logical Inference in Large Language Model Reasoning [49.58786377307728]
  This paper adopts an exploratory approach by introducing a controlled evaluation environment for analogical reasoning. We analyze the comparative dynamics of inductive, abductive, and deductive inference pipelines. We investigate advanced paradigms such as hypothesis selection, verification, and refinement, revealing their potential to scale up logical inference.
  arXiv  Detail & Related papers  (2025-02-16T15:54:53Z)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models [88.84491005030316]
  Generative training has enabled Vision-Language Models (VLMs) to tackle various complex tasks. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval. This paper proposes a unified approach that integrates the strengths of both paradigms.
  arXiv  Detail & Related papers  (2024-11-01T01:51:31Z)
• LLMTemporalComparator: A Tool for Analysing Differences in Temporal Adaptations of Large Language Models [17.021220773165016]
  This study addresses the challenges of analyzing temporal discrepancies in large language models (LLMs) trained on data from different time periods. We propose a novel system that compares in a systematic way the outputs of two LLM versions based on user-defined queries.
  arXiv  Detail & Related papers  (2024-10-05T15:17:07Z)
• Unraveling Text Generation in LLMs: A Stochastic Differential Equation Approach [3.4039202831583903]
  This paper explores the application of Differential Equations (SDE) to interpret the text generation process of Large Language Models (LLMs) such as GPT-4. We represent this generation process using SDE to capture both deterministic trends and perturbations. We fit these functions using neural networks and validate the model on real-world text corpora.
  arXiv  Detail & Related papers  (2024-08-17T15:30:27Z)
• Semantic Change Characterization with LLMs using Rhetorics [0.1474723404975345]
  We investigate the potential of LLMs in characterizing three types of semantic change: thought, relation, and orientation. Our results highlight the effectiveness of LLMs in capturing and analyzing semantic changes, providing valuable insights to improve computational linguistic applications.
  arXiv  Detail & Related papers  (2024-07-23T16:32:49Z)
• LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments [70.91258869156353]
  We introduce LangSuitE, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuitE offers adaptability to diverse environments without multiple simulation engines. We devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information.
  arXiv  Detail & Related papers  (2024-06-24T03:36:29Z)
• When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges [50.280704114978384]
  Pre-trained large language models (LLMs) exhibit powerful capabilities for generating natural text. Evolutionary algorithms (EAs) can discover diverse solutions to complex real-world problems.
  arXiv  Detail & Related papers  (2024-01-19T05:58:30Z)
• In-Context Learning Dynamics with Random Binary Sequences [16.645695664776433]
  We propose a framework that enables us to analyze in-context learning dynamics. Inspired by the cognitive science of human perception, we use random binary sequences as context. In the latest GPT-3.5+ models, we find emergent abilities to generate seemingly random numbers and learn basic formal languages.
  arXiv  Detail & Related papers  (2023-10-26T17:54:52Z)
• Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning [57.74233319453229]
  Large language models (LLMs) have emerged as a groundbreaking technology and their unparalleled text generation capabilities have sparked interest in their application to the fundamental sentence representation learning task. We propose MultiCSR, a multi-level contrastive sentence representation learning framework that decomposes the process of prompting LLMs to generate a corpus. Our experiments reveal that MultiCSR enables a less advanced LLM to surpass the performance of ChatGPT, while applying it to ChatGPT achieves better state-of-the-art results.
  arXiv  Detail & Related papers  (2023-10-17T03:21:43Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)
• APo-VAE: Text Generation in Hyperbolic Space [116.11974607497986]
  In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adrial Poincare Variversaational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. Experiments in language modeling and dialog-response generation tasks demonstrate the winning effectiveness of the proposed APo-VAE model.
  arXiv  Detail & Related papers  (2020-04-30T19:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.