Intention Collapse: Intention-Level Metrics for Reasoning in Language Models

• URL: http://arxiv.org/abs/2601.01011v1
• Date: Sat, 03 Jan 2026 00:19:53 GMT
• Title: Intention Collapse: Intention-Level Metrics for Reasoning in Language Models
• Abstract summary: We call this process intention collapse: a many-to-one projection from a high dimensional intention space I into an external language space L.<n>We define three model agnostic intention metrics (intention entropy, effective dimensionality dimeff, and latent knowledge recoverability Recov)<n>Using a 4 bit Mistral 7B model on 200 GSM8K problems, we compare a direct answer baseline, a chain of thought (CoT) regime, and a babble control.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Every act of language generation compresses a rich internal state into a single token sequence. We call this process intention collapse: a many-to-one projection from a high dimensional intention space I into an external language space L. We formalize intention collapse for contemporary language models, define three simple, model agnostic intention metrics (intention entropy Hint, effective dimensionality dimeff, and latent knowledge recoverability Recov), and propose an empirical agenda for studying how inference time computation shapes internal intentions before they are verbalized. We also report a first small scale experiment. Using a 4 bit Mistral 7B model on 200 GSM8K problems, we compare a direct answer baseline, a chain of thought (CoT) regime, and a babble control. CoT raises accuracy from 5.5 percent to 53 percent, sharply reduces pre collapse intention entropy (from 1.42 to 0.37 bits), and shows higher global effective dimensionality than the other regimes despite producing fewer tokens than babble. At the same time, Hint has little item level predictive power, and a linear probe on I achieves AUROC 0.65 in the CoT regime but only about chance in the baseline regime, where it collapses to the majority class. These preliminary results indicate that intention level metrics can distinguish inference regimes and expose latent information that is partly lost during collapse, while also revealing important limitations of our current proxies

Related papers

• Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs [0.0]
  I introduce J-PragEval-v0, a minimal-pair benchmark isolating four phenomena from surface fluency.<n>I combine it with linear probes and teacher-forced log-probability evaluation to ask where inside TinySwallow-1.5B the corresponding contrasts live.
  arXiv  Detail & Related papers  (2026-08-14T21:04:21Z)
• Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents [22.30468215041597]
  A tool-using agent is given the same task in a different language, does it still take the same steps?<n>We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages.
  arXiv  Detail & Related papers  (2026-08-11T16:18:34Z)
• Why Large Language Models Fail at Tabular Prediction [1.1908340728628395]
  Large language models (LLMs) have become the default tool for a remarkable range of tasks.<n>But they have had conspicuously little success at one of the most common machine learning workloads.
  arXiv  Detail & Related papers  (2026-08-03T15:52:51Z)
• PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models [23.263895549689863]
  Existing methods rely solely on next-token prediction loss, using only language tokens for supervision.<n>mname explicitly supervises intermediate point cloud tokens to preserve fine-grained 3D geometric-semantic information.<n>Experiments on ModelNet40 and averse datasets demonstrate that our method achieves textbf2.08 pp improvement on average for classification tasks.
  arXiv  Detail & Related papers  (2026-02-28T02:17:46Z)
• Agentic World Modeling for 6G: Near-Real-Time Generative State-Space Reasoning [70.56067503630486]
  We argue that sixth-generation (6G) intelligence is not fluent token prediction but calibrated the capacity to imagine and choose.<n>We show that WM-MS3M cuts mean absolute error (MAE) by 1.69% versus MS3M with 32% fewer parameters and similar latency, and achieves 35-80% lower root mean squared error (RMSE) than attention/hybrid baselines with 2.3-4.1x faster inference.
  arXiv  Detail & Related papers  (2025-11-04T17:22:22Z)
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought [23.847410628315544]
  We introduce **Language-Mixed CoT**, a reasoning schema that switches between English and a target language.<n>We train ninve models (4B-35B) across six families (Qwen2.5, Llama-3.1, Gemma-3, etc)<n>Our best model, **KO-REAson-35B**, achieves state-of-the-art performance, with the highest overall average score (64.0 pm 25)
  arXiv  Detail & Related papers  (2025-10-05T14:39:41Z)
• Thought calibration: Efficient and confident test-time scaling [11.028893528095196]
  Reasoning large language models achieve impressive test-time scaling by thinking for longer, but this performance gain comes at significant compute cost.<n>We propose thought calibration to decide dynamically when thinking can be terminated.<n>We realize this framework through lightweight probes that operate on top of the language model's hidden representations.
  arXiv  Detail & Related papers  (2025-05-23T22:17:18Z)
• Understanding In-context Learning of Addition via Activation Subspaces [73.8295576941241]
  We study a structured family of few-shot learning tasks for which the true prediction rule is to add an integer $k$ to the input.<n>We then perform an in-depth analysis of individual heads, via dimensionality reduction and decomposition.<n>Our results demonstrate how tracking low-dimensional subspaces of localized heads across a forward pass can provide insight into fine-grained computational structures in language models.
  arXiv  Detail & Related papers  (2025-05-08T11:32:46Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• The Remarkable Robustness of LLMs: Stages of Inference? [5.346230590800585]
  We investigate the robustness of Large Language Models (LLMs) to structural interventions by deleting and swapping adjacent layers during inference.<n>Surprisingly, models retain 72-95% of their original top-1 prediction accuracy without any fine-tuning.
  arXiv  Detail & Related papers  (2024-06-27T17:57:03Z)
• Simple linear attention language models balance the recall-throughput tradeoff [60.06020449520365]
  We propose BASED, a simple architecture combining linear and sliding window attention.<n>We train language models up to 1.3b parameters and show that BASED matches the strongest sub-quadratic models in perplexity and outperforms them on real-world recall-intensive tasks by 6.22 accuracy points.
  arXiv  Detail & Related papers  (2024-02-28T19:28:27Z)
• An Analysis and Mitigation of the Reversal Curse [70.13419502543915]
  Recent research observed a noteworthy phenomenon in large language models (LLMs) The reversal curse is that when dealing with two entities, $a$ and $b$, LLMs excel in handling sequences in the form of $aRb$,'' but encounter challenges when processing $bR-1a$''
  arXiv  Detail & Related papers  (2023-11-13T17:01:12Z)
• Uncertainty-Aware Unlikelihood Learning Improves Generative Aspect Sentiment Quad Prediction [52.05304897163256]
  We propose a template-agnostic method to control the token-level generation. Specifically, we introduce Monte Carlo dropout to understand the built-in uncertainty of pre-trained language models. We further propose marginalized unlikelihood learning to suppress the uncertainty-aware mistake tokens.
  arXiv  Detail & Related papers  (2023-06-01T07:49:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.