Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes

• URL: http://arxiv.org/abs/2603.04426v1
• Date: Mon, 16 Feb 2026 23:35:29 GMT
• Title: Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes
• Abstract summary: Crosscoders learn shared dictionaries of interpretable latent directions between base and fine-tuned models.<n>We introduce Delta-Crosscoder, which combines BatchTopK sparsity with a delta-based loss prioritizing directions that change between models.
• Score: 16.269397514663417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model diffing methods aim to identify how fine-tuning changes a model's internal representations. Crosscoders approach this by learning shared dictionaries of interpretable latent directions between base and fine-tuned models. However, existing formulations struggle with narrow fine-tuning, where behavioral changes are localized and asymmetric. We introduce Delta-Crosscoder, which combines BatchTopK sparsity with a delta-based loss prioritizing directions that change between models, plus an implicit contrastive signal from paired activations on matched inputs. Evaluated across 10 model organisms, including synthetic false facts, emergent misalignment, subliminal learning, and taboo word guessing (Gemma, LLaMA, Qwen; 1B-9B parameters), Delta-Crosscoder reliably isolates latent directions causally responsible for fine-tuned behaviors and enables effective mitigation, outperforming SAE-based baselines, while matching the Non-SAE-based. Our results demonstrate that crosscoders remain a powerful tool for model diffing.

Related papers

• DIFF-ERO: A Conformance-Aware Loss for Deep Learning in Process Mining [8.304025820176394]
  We introduce DIFF-ERO, a conformance-aware loss function for deep learning models on process data.<n>DIFF-ERO is a differentiable formulation of entropy-based conformance that incorporates control-flow information during training.<n>We show improved predictive performance where structure matters most while maintaining parity elsewhere.
  arXiv  Detail & Related papers  (2026-06-12T09:06:05Z)
• Memory by Design: Probabilistic Sequence Layers [19.16406809753255]
  We introduce a way to derive efficient recurrent sequence maps from explicit assumptions about memory.<n>A design model writes evidence into memory by exact Bayesian filtering.<n>A query-dependent readout produces a predictive distribution whose mean is the layer output.
  arXiv  Detail & Related papers  (2026-05-29T11:13:00Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• Delta-K: Boosting Multi-Instance Generation via Cross-Attention Augmentation [3.27529630826945]
  Diffusion Models often suffer from concept omission when synthesizing complex scenes.<n>We propose Delta-K, a backbone-agnostic and plug-and-play inference framework that tackles omission by operating directly in the shared cross-attention Key space.
  arXiv  Detail & Related papers  (2026-03-10T20:23:00Z)
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model [72.73512218682187]
  We introduce ReDiff, a refining-enhanced diffusion framework that teaches the model to identify and correct its own errors.<n>Our approach features a two-stage training process: first, we instill a foundational revision capability by training the model to revise synthetic errors; second, we implement a novel online self-correction loop.<n>This mistake-driven learning endows the model with the crucial ability to revisit and refine its already generated output, effectively breaking the error cascade.
  arXiv  Detail & Related papers  (2025-10-22T06:58:55Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Rethinking Layer-wise Model Merging through Chain of Merges [21.26982153528304]
  Chain of Merges (CoM) is a layer-wise merging procedure that sequentially merges weights across layers while sequentially updating activation statistics.<n> Experiments on standard benchmarks demonstrate that CoM achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-29T08:44:47Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning [9.761667621068787]
  Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented as latent directions in both the base and fine-tuned models.<n>We identify two issues which stem from the crosscoders L1 training loss that can misattribute concepts as unique to the fine-tuned model, when they really exist in both models.<n>We train a crosscoder with BatchTopK loss and show that it substantially mitigates these issues, finding more genuinely chat-specific and highly interpretable concepts.
  arXiv  Detail & Related papers  (2025-04-03T17:50:24Z)
• Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflicts [14.563722352134949]
  Transformer-based pre-trained models have achieved great improvements in semantic matching. Existing models still suffer from insufficient ability to capture subtle differences. We propose a novel Dual Path Modeling Framework to enhance the model's ability to perceive subtle differences.
  arXiv  Detail & Related papers  (2023-02-24T09:29:55Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)
• Do Generative Models Know Disentanglement? Contrastive Learning is All You Need [59.033559925639075]
  We propose an unsupervised and model-agnostic method: Disentanglement via Contrast (DisCo) in the Variation Space. DisCo achieves the state-of-the-art disentanglement given pretrained non-disentangled generative models, including GAN, VAE, and Flow.
  arXiv  Detail & Related papers  (2021-02-21T08:01:20Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.