Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits

• URL: http://arxiv.org/abs/2607.01940v1
• Date: Thu, 02 Jul 2026 09:32:57 GMT
• Title: Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits
• Abstract summary: Conditional Co-Ablation asks how much each remaining unit's ablation effect grows once a primary set has been removed.<n>On the GPT-2-small IOI circuit, CoAx raises backup-head recovery from 0.33 to 0.91 ROC-AUC.
• Score: 47.668752416295185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mechanistic interpretability often relies on component-level interventions to discover how a model produces a behavior. This guides attribution, capability knockout, and model pruning downstream to operate by scoring each unit by the effect of ablation in isolation. Such first-order scoring is natural when component importance is additive, but becomes misleading when a transformer self-repairs: after a primary component is removed, a dormant backup can take over, muting the primary's measured effect while the backup itself appears irrelevant on the intact model. We recast this failure as a recovery task, conditional circuit completion, and introduce Conditional Co-Ablation (CoAx), a label-free, output-grounded score that asks how much each remaining unit's ablation effect grows once a primary set has been removed. This conditional growth exposes the second-order interaction that single-unit scores discard. On the GPT-2-small IOI circuit, CoAx raises backup-head recovery from 0.33 to 0.91 ROC-AUC, outperforming all baselines, including self-repair-aware gradient scores (best 0.82); counterfactual patching verifies that the recovered heads causally carry the repair. The same label-free procedure transfers to induction across eight models. Beyond discovery, the recovered backups correct self-repair-masked attribution, identify the components required for capability knockout, and yield repair-aware structured pruning scaling from 124M to 7B. Component importance is therefore not merely an isolated-unit property: in robust circuits, the components that matter can become visible only under the interventions that make them necessary.

Related papers

• ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers [58.411900827911346]
  We show that the missing ingredient is the treatment of residual connections.<n>We introduce Residual-aware Layer-wise Relevance Propagation (ResLRP)<n>ResLRP is a simple extension of LRP whose propagation rules explicitly account for cancellations in residual branches.
  arXiv  Detail & Related papers  (2026-09-15T13:17:30Z)
• EvoUndo: Recoverability-Constrained Self-Evolution for LLM Agent Harnesses [0.0]
  We introduce EvoUndo, a framework for representing, synthesizing, diagnosing, and independently verifying recoverability of model-generated self-modifications.<n>Across 600 unseen one-shot self-evolution tasks, we identify 197 capability-improving mutations that fail recoverability verification.
  arXiv  Detail & Related papers  (2026-08-28T14:15:36Z)
• Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions [87.95757610455173]
  We introduce an injectively aligned leave-one-cell-out protocol over support x operation grids and SO-OPF.<n>We separate two questions that aggregate scores conflate: whether the carrier composes held-out bindings when the grid is known, and whether that grid can be recovered from flat cell labels.
  arXiv  Detail & Related papers  (2026-08-06T15:38:55Z)
• Bayesian Repetition Penalty: A Principled Adjacent-Conditional Framework for Reversing Attention Collapse in Autoregressive Language Models [9.240704665480939]
  We present a principled framework that penalises or compensates anomalous confidence arising from collapsed generation patterns.<n>The resulting self-normalising penalty ratio $R=f(m,n,p)/f(np,n,p)$ requires no ad hoc standardisation and admits a closed-form logit offset with zero approximation error.
  arXiv  Detail & Related papers  (2026-07-17T17:01:57Z)
• CURE: Controllable Unified Image Restoration for Complex Degradations [11.660815850318114]
  CURE is a unified framework that enables controllable restoration in complex degradation settings.<n>CURE delivers state-of-the-art performance on composite degradation benchmarks.
  arXiv  Detail & Related papers  (2026-07-03T07:34:11Z)
• The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching [28.393865595242872]
  Activation patching is the primary tool in mechanistic interpretability.<n>It attributes causal responsibility for a model behavior to each of its individual components by estimating its natural indirect effect.<n>It also contains interaction effects (INT) that measure how much the component's causal effect itself depends on the state of other components in the model.
  arXiv  Detail & Related papers  (2026-06-25T19:52:16Z)
• SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior [38.75847400495247]
  Sparse Autoencoders (SAEs) decompose residual-stream activations into interpretable features.<n>We show that clamping a specific harmful feature is expected to reliably prevent model misbehavior.<n>We formulate this vulnerability as post-intervention recovery, a constrained residual-space optimization problem.
  arXiv  Detail & Related papers  (2026-06-16T15:04:17Z)
• Deployment-Time Memorization in Foundation-Model Agents [18.40535884093806]
  We study agent memory as a privacy-utility frontier measured by Personalization Recall (PR) and Adrial Extraction Rate (AER)<n>Key-fact summarization reduces canary extraction by 76% on Gemma 3 12B and 64% on GPT-4o-mini while preserving nearly all personalization recall.<n>On LongMemEval, key-fact summarization reduces canary extraction by 76% on Gemma 3 12B and 64% on GPT-4o-mini while preserving nearly all personalization recall.
  arXiv  Detail & Related papers  (2026-06-08T18:38:41Z)
• Beyond Spatial Compression: Interface-Centric Generative States for Open-World 3D Structure [55.17771962316751]
  Current 3D tokenizers treat representation as spatial compression, but leave component ownership and attachment validity implicit.<n>We formulate an alternative view, interface-centric generative states, in which tokenization constructs an operational state rather than a passive compressed code.<n>We instantiate this formulation with Component-Conditioned Canonical Local Tokens (C2LT-3D), factorizing representation into canonical local geometry, partition-conditioned context, and relational seam variables.
  arXiv  Detail & Related papers  (2026-05-11T12:09:28Z)
• ANNEAL: Adapting LLM Agents via Governed Symbolic Patch Learning [12.44509691077682]
  We introduce ANNEAL, a neuro-symbolic agent that converts recurring failures into governed symbolic edits of a process knowledge graph.<n>Its core mechanism, Failure-Driven Knowledge Acquisition (FDKA), localizes the responsible operator, synthesizes a typed patch through constrained LLM generation, and validates the proposal.<n>Across four domains and 27 multi-seed runs, ANNEAL is the only evaluated system that commits persistent structural repairs.
  arXiv  Detail & Related papers  (2026-05-04T05:24:03Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation [0.996038809132083]
  We introduce CUE-R, a framework for measuring per-evidence-item operational utility in single-shot RAG.<n>We show that CUE-R perturbs individual evidence items via REMOVE, REPLACE, and DUPLICATE operators, then measures changes along three utility axes.<n>Our results suggest that answer-only evaluation misses important evidence effects and that intervention-based utility analysis is a practical complement for RAG evaluation.
  arXiv  Detail & Related papers  (2026-04-07T06:05:08Z)
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training [86.64715217940274]
  Outliers function jointly with normalization.<n>Outliers serve more as rescale factors rather than contributors.<n>Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling.
  arXiv  Detail & Related papers  (2026-01-30T13:29:45Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• REFINE: Inversion-Free Backdoor Defense via Model Reprogramming [60.554146386198376]
  Backdoor attacks on deep neural networks (DNNs) have emerged as a significant security threat.<n>We propose REFINE, an inversion-free backdoor defense method based on model reprogramming.
  arXiv  Detail & Related papers  (2025-02-22T07:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.