Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolution

• URL: http://arxiv.org/abs/2608.24460v1
• Date: Tue, 25 Aug 2026 12:09:19 GMT
• Title: Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolution
• Abstract summary: We train transformers on a synthetic language where recency and rarity are exactly coextensive.<n>We separate them with a minimal causal edit that inverts one cue while holding the truth, token count and answer position fixed.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a context asserts two values for one fact, a model commits to a cue -- recency, repetition, position -- but natural data rarely makes these disagree, so behavior cannot reveal which. We train 26M-parameter transformers on a synthetic language where recency and rarity are exactly coextensive, and separate them with a minimal causal edit that inverts one cue while holding the truth, token count and answer position fixed. All 75 runs reach accuracy >= 0.999, including where the trivial heuristic fails, so no held-in evaluation distinguishes them. Under intervention the per-cell readout does not replicate: 13 of 25 cells differ by more than 0.3 in sign fraction across three seeds, the largest by 0.879 against a standard error of 0.025. The construction predicts this -- coextensive rules leave the objective indifferent between them -- and the variance is ordered by how much of the optimization each comparison releases. What replicates is timing: escape from a positional shortcut with a closed-form ceiling, monotone in redundancy. Probed before that escape, attribution reverses sign in 32 of 75 runs at unchanged accuracy, and gating on circuit formation is necessary but not sufficient. The corpus fixes when a mechanism appears, not which one -- a criterion for when mechanistic attribution to data is available at all, and our construction makes the unavailable case exact.

Related papers

• Cross-Block Conditioning in Deep Boltzmann Machines for Statistical Data Fusion [0.0]
  We propose observed-block multi-prediction, which restricts the multi-prediction objective to targets drawn from what each row actually observes.<n>Having a discriminative criterion that survives the setting lets us ask whether the joint model is needed at all.
  arXiv  Detail & Related papers  (2026-09-14T02:26:16Z)
• A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram [0.0]
  Sparse autoencoders (SAEs) are increasingly used to recover interpretable features from neural-network activations.<n>The MAIS-O43 open problem proposes a controlled experiment to characterize when recovery of a true synthetic dictionary gives way to feature merging.
  arXiv  Detail & Related papers  (2026-09-09T15:13:42Z)
• Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores [31.709549159768727]
  We observe a consistent readout gap across diverse reasoning benchmarks.<n> hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses.<n>Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic.
  arXiv  Detail & Related papers  (2026-08-31T16:43:55Z)
• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection [0.0]
  Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings.<n>We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against.
  arXiv  Detail & Related papers  (2026-08-12T23:27:20Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop [0.0]
  We study an adversarial test-hardening loop under a mechanical oracle.<n>A Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those.<n>We release both protocols, all receipts, and the analysis code.
  arXiv  Detail & Related papers  (2026-07-25T02:38:07Z)
• Memoir: Should a Model Write to Its Memory While It Thinks? [0.0]
  Memoir combines per-sample fast memory, shared slow parameters, variable-depth latent recurrence, and a future-latent energy objective.<n>We compare a coupled arm against an otherwise identical read-only pondering arm.<n>A predicted failure in which memory rewriting corrupts the energy signal did not occur.
  arXiv  Detail & Related papers  (2026-07-22T23:34:56Z)
• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary [0.0]
  Can a language model read the quality of its ongoing computation?<n>Can an external intervention turn that readout into better outcomes?<n>We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT.
  arXiv  Detail & Related papers  (2026-07-20T22:40:36Z)
• Forgetting Is Not a Fix: Path Dependence in Sequential Engram Editing [35.76482964927589]
  We test the hypothesis that concept-specific memory traces become linear objects that can be extracted once and combined arithmetically.<n>We run that test on the authors' own reference implementation, at their reported best edit strength.<n>For unlearning-as-compliance: erasure certified today does not certify the artifact after its next edit.
  arXiv  Detail & Related papers  (2026-07-06T23:45:06Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• PBT-Bench: Benchmarking AI Agents on Property-Based Testing [29.035258104995204]
  We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.<n>Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them.<n>Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%.
  arXiv  Detail & Related papers  (2026-05-13T18:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.