When and How Long? The Readout-Mediator Angle in Temporal Reasoning

• URL: http://arxiv.org/abs/2605.29126v1
• Date: Wed, 27 May 2026 21:38:17 GMT
• Title: When and How Long? The Readout-Mediator Angle in Temporal Reasoning
• Abstract summary: linear probe can decode a representation almost perfectly and yet be completely irrelevant to how the model uses it.<n> ablating a four-dimensional subspace found by Distributed Alignment Search (DAS) at the same layer collapses performance entirely.<n>We reverse-engineering the circuit reveals why: attention heads month-grained context through learned QK offsets at $pm30$ and $pm61$ days.
• Score: 1.0195618602298682
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: A linear probe can decode a representation almost perfectly and yet be completely irrelevant to how the model uses it. On calendar-date duration reasoning in language models, a $\sin$/$\cos$ probe recovers day-of-year from a layer's activations, yet ablating its direction has no effect on the model's answers -- while ablating a four-dimensional subspace found by Distributed Alignment Search (DAS) at the same layer collapses performance entirely. We measure the angle between these two subspaces -- the \emph{readout-mediator angle} -- and find it indistinguishable from the angle between two random subspaces (the Haar-uniform null), meaning the probe has learned a direction orthogonal to the model's actual computation. Reverse-engineering the circuit reveals why: attention heads route month-grained context through learned QK offsets at ${\pm}30$ and ${\pm}61$ days, and MLPs then convert \emph{when} (absolute date) into \emph{how long} (duration) -- all downstream of the causal subspace the probe never touches. Sparse-autoencoder decomposition confirms the split: probe-aligned and DAS-aligned features encode semantically disjoint concepts with negligible causal overlap. The dissociation replicates across four scales ($1.5$-$9\,$B) and two model families, with preliminary evidence on two further domains (spatial displacement, symbolic arithmetic), suggesting that readout-mediator orthogonality is a general failure mode of probe-based interpretability. This directly undermines proposals to deploy probes as runtime safety monitors: the probe can report high confidence on a direction the model has silently abandoned.

Related papers

• Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails [22.49571926071395]
  We show that pairing a token's forward state against a emphcandidate's backward state (cross-direction'' pairing, $F_i$ vs. $B_j$) consistently emphunderperforms same-direction pairing, and the penalty emphgrows, not shrinks, with token distance, both paired-bootstrap significant.
  arXiv  Detail & Related papers  (2026-08-21T00:50:25Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training [56.323119575322146]
  Extra-Merge consistently outperforms standard merging baselines in experiments across GPT-2 and LLaMA families.<n>It yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon citepjordan2024muon.
  arXiv  Detail & Related papers  (2026-05-26T02:48:34Z)
• Metric--Phase Fields: Decoupling Distance and Sign for Thin-Structure Reconstruction from Unoriented Point Clouds [67.30236447078187]
  We introduce Metric-Phase Fields (MPFs), a decoupled implicit representation that separates metric proximity from topological phase.<n>Experiments on both synthetic and scanned thin-shell and thin-plate shapes demonstrate that MPFs preserve thin and layered structures more faithfully than recent SDF-based methods.
  arXiv  Detail & Related papers  (2026-05-25T07:08:57Z)
• Reading Calibrated Uncertainty from Language Model Trajectories [46.663987199083245]
  Methods that probe the model's internal activations feed raw hidden states into opaque snapshots and leave implicit the layer-wise trajectory by which a representation is formed.<n>We extract eleven scale-invariant geometric features, tracing the cumulative path of per-layer updates, and feed them to a sparse linear probe.<n>The probe outperforms MSP under selective abstention, with baseline scaling with up to 21 AURC points.
  arXiv  Detail & Related papers  (2026-05-19T19:24:29Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• Temporal Memory for Resource-Constrained Agents: Continual Learning via Stochastic Compress-Add-Smooth [0.0]
  An agent that operates sequentially must incorporate new experience without forgetting old experience, under a fixed memory budget.<n>We propose a framework in which memory is not a parameter vector but a process: a Diffusion Bridge on a replay interval.<n>The framework provides a fully analytical Ising model'' of continual learning in which the mechanism, rate, and form of forgetting can be studied with mathematical precision.
  arXiv  Detail & Related papers  (2026-03-31T12:04:11Z)
• In Machina N400: Pinpointing Where a Causal Language Model Detects Semantic Violations [0.11458853556386796]
  We evaluate the causal language model (phi-2) using a carefully curated corpus, with sentences that concluded plausibly or implausibly.<n>To investigate how violations are encoded, we conducted a per-layer detection using a linear probe.
  arXiv  Detail & Related papers  (2025-11-24T15:43:56Z)
• Coupled Data and Measurement Space Dynamics for Enhanced Diffusion Posterior Sampling [27.146380722473932]
  Inverse problems, where the goal is to recover an unknown signal from noisy or incomplete measurements, are central to medical imaging, remote sensing, and computational biology.<n>We propose a novel framework called emphcoupled data and space diffusion posterior sampling (C-DPS), which eliminates the need for constraint tuning or likelihood measurement.<n>C-DPS consistently outperforms existing baselines, both qualitatively and quantitatively, across multiple inverse problem benchmarks.
  arXiv  Detail & Related papers  (2025-10-08T18:59:16Z)
• Monge-Ampere Regularization for Learning Arbitrary Shapes from Point Clouds [69.69726932986923]
  We propose the scaled-squared distance function (S$2$DF), a novel implicit surface representation for modeling arbitrary surface types.<n>S$2$DF does not distinguish between inside and outside regions while effectively addressing the non-differentiability issue of UDF at the zero level set.
  arXiv  Detail & Related papers  (2024-10-24T06:56:34Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Analytic Signal Phase in $N-D$ by Linear Symmetry Tensor--fingerprint modeling [69.35569554213679]
  We show that the Analytic Signal phase, and its gradient have a hitherto unstudied discontinuity in $2-D $ and higher dimensions. This shortcoming can result in severe artifacts whereas the problem does not exist in $1-D $ signals. We suggest the use of Linear Symmetry phase, relying on more than one set of Gabor filters, but with a negligible computational add-on.
  arXiv  Detail & Related papers  (2020-05-16T21:17:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.