Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs

• URL: http://arxiv.org/abs/2602.11729v1
• Date: Thu, 12 Feb 2026 08:53:25 GMT
• Title: Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs
• Abstract summary: Crosscoders are one solution capable of cross-architecture model diffing but have only ever been applied to base vs finetune comparisons.<n>We introduce Dedicated Feature Crosscoders (DFCs), an architectural modification designed to better isolate features unique to one model.<n>Using this technique, we find in an unsupervised fashion features including Chinese Communist Party alignment in Qwen3-8B and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B.
• Score: 4.8608047541422925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model diffing, the process of comparing models' internal representations to identify their differences, is a promising approach for uncovering safety-critical behaviors in new models. However, its application has so far been primarily focused on comparing a base model with its finetune. Since new LLM releases are often novel architectures, cross-architecture methods are essential to make model diffing widely applicable. Crosscoders are one solution capable of cross-architecture model diffing but have only ever been applied to base vs finetune comparisons. We provide the first application of crosscoders to cross-architecture model diffing and introduce Dedicated Feature Crosscoders (DFCs), an architectural modification designed to better isolate features unique to one model. Using this technique, we find in an unsupervised fashion features including Chinese Communist Party alignment in Qwen3-8B and Deepseek-R1-0528-Qwen3-8B, American exceptionalism in Llama3.1-8B-Instruct, and a copyright refusal mechanism in GPT-OSS-20B. Together, our results work towards establishing cross-architecture crosscoder model diffing as an effective method for identifying meaningful behavioral differences between AI models.

Related papers

• Beyond Scale and Generation: Understanding Language Model-based Entity Matching [18.904187328683513]
  We show that model variant is critical for bi-encoders.<n>Generative matchers do not universally outperform cross-encoders.<n> larger models rely more heavily on shortcut learning.
  arXiv  Detail & Related papers  (2026-07-27T17:29:18Z)
• Differentiable Faithfulness Alignment for Cross-Model Circuit Transfer [73.0825876244911]
  We introduce DFA, a framework that transfers circuit information from a smaller source model to a larger target model.<n>We evaluate DFA on Llama-3 and Qwen-2.5 across six tasks spanning factual retrieval, multiple-choice reasoning, and arithmetic.<n>DFA consistently outperforms simple baselines and, in some settings, recovers target-model circuits with faithfulness comparable to or stronger than direct attribution.
  arXiv  Detail & Related papers  (2026-04-27T10:49:10Z)
• AutoMerge: Search-Based Model Merging Framework for Effective Model Reuse [8.950520457150178]
  Recently, model merging has emerged in the domain of large language models (LLMs) as a training-free approach.<n>No prior work has systematically investigated whether such an approach can be effectively applied to other deep learning models.<n>We present the first systematic study that evaluates five model merging techniques on three distinct model architectures.
  arXiv  Detail & Related papers  (2026-01-30T09:27:01Z)
• Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEs [27.331524018411926]
  We compare encoder-only and decoder-only models on cross-modal adaptation for time-dependent simulation tasks.<n>We find that decoder-only models are far worse than encoder-only models, when existing approaches are applied unmodified.<n>We introduce two novel approaches, Parallel Flipping and Sequence Doubling, attempting to mimic bidirectionality in autoregressive models.
  arXiv  Detail & Related papers  (2025-10-06T18:46:50Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretability [9.90112908284836]
  We introduce (Sparse Autoencoders for Aligned Representation of Concepts) a new framework that learns a single, unified latent space shared across diverse architectures.<n>On Open Images, dramatically improves concept alignment, achieving a Jaccard similarity of 0.80, more than tripling the alignment compared to previous methods.
  arXiv  Detail & Related papers  (2025-07-07T22:29:00Z)
• Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture [65.88390432432116]
  Masked diffusion models (MDMs) are emerging as viable alternatives to autoregressive (AR) models.<n>AR models are often decoder-only, while MDMs have largely been encoder-only.<n>This research evaluates MDMs within a decoder-only framework to:.<n>Investigate architectural influences (decoder-only vs. encoder-only) within MDMs.
  arXiv  Detail & Related papers  (2025-06-24T18:22:25Z)
• Cross-architecture universal feature coding via distribution alignment [88.73189953617594]
  We introduce a new research problem: cross-architecture universal feature coding (CAUFC)<n>We propose a two-step distribution alignment method. First, we design the format alignment method that CNN and Transformer features into a consistent 2D token format. Second, we propose the feature value alignment method that harmonizes statistical distributions via truncation and normalization.<n>As a first attempt to study CAUFC, we evaluate our method on the image classification task. Experimental results demonstrate that our method achieves superior rate-accuracy trade-offs compared to the architecture-specific baseline.
  arXiv  Detail & Related papers  (2025-06-15T06:14:02Z)
• Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning [9.761667621068787]
  Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented as latent directions in both the base and fine-tuned models.<n>We identify two issues which stem from the crosscoders L1 training loss that can misattribute concepts as unique to the fine-tuned model, when they really exist in both models.<n>We train a crosscoder with BatchTopK loss and show that it substantially mitigates these issues, finding more genuinely chat-specific and highly interpretable concepts.
  arXiv  Detail & Related papers  (2025-04-03T17:50:24Z)
• MI-DETR: An Object Detection Model with Multi-time Inquiries Mechanism [67.56918651825056]
  We propose a new decoder architecture with the parallel Multi-time Inquiries (MI) mechanism.<n>Our MI based model, MI-DETR, outperforms all existing DETR-like models on COCO benchmark.<n>A series of diagnostic and visualization experiments demonstrate the effectiveness, rationality, and interpretability of MI.
  arXiv  Detail & Related papers  (2025-03-03T12:19:06Z)
• Switchable Representation Learning Framework with Self-compatibility [50.48336074436792]
  We propose a Switchable representation learning Framework with Self-Compatibility (SFSC) SFSC generates a series of compatible sub-models with different capacities through one training process. SFSC achieves state-of-the-art performance on the evaluated datasets.
  arXiv  Detail & Related papers  (2022-06-16T16:46:32Z)
• Slimmable Domain Adaptation [112.19652651687402]
  We introduce a simple framework, Slimmable Domain Adaptation, to improve cross-domain generalization with a weight-sharing model bank. Our framework surpasses other competing approaches by a very large margin on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T06:28:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.