Characterizing Linear Alignment Across Language Models

• URL: http://arxiv.org/abs/2603.18908v3
• Date: Thu, 26 Mar 2026 15:17:05 GMT
• Title: Characterizing Linear Alignment Across Language Models
• Abstract summary: Language models increasingly appear to learn similar representations, despite differences in training objectives, architectures, and data modalities.<n>This capability unlocks new potential application domains, such as settings where security, privacy, or competitive constraints prohibit direct data or model sharing.
• Score: 10.66607150500579
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models increasingly appear to learn similar representations, despite differences in training objectives, architectures, and data modalities. This emerging compatibility between independently trained models introduces new opportunities for cross-model alignment to downstream objectives. Moreover, this capability unlocks new potential application domains, such as settings where security, privacy, or competitive constraints prohibit direct data or model sharing. In this work, we investigate the extent to which representational convergence enables practical linear alignment between large language models. Specifically, we learn affine transformations between the final hidden states of independent models and empirically evaluate these mappings across text generation, embedding classification, and out-of-distribution detection. We find that performance is largely preserved across model pairs, and show for the first time that linear alignment sometimes enables text generation across independently trained models. We further highlight a potential application of linear alignment for privacy-preserving cross-silo inference. The framework learns an affine transformation over a shared public dataset and uses homomorphic encryption to protect client queries. By encrypting only the linear classification operation, the method achieves sub-second inference latency.

Related papers

• One Framework for All: Cross-Modal Membership Inference for Generative Models [16.609636193065377]
  We present the first comprehensive study of a unified membership inference framework that applies across text-to-text, text-to-image, and image-to-text modalities.<n>Our approach is grounded in a key modality-agnostic observation: the output distribution of a generative model can approximate its training data distribution.
  arXiv  Detail & Related papers  (2026-07-05T14:45:29Z)
• Discriminative Flow Matching Via Local Generative Predictors [0.0]
  We propose a framework that reformulates classification and object detection as a conditional transport process.<n>Our method attaches multiple independent flow predictors to a shared backbone.<n>This architecture provides the flexibility to update blocks either sequentially to minimise activation memory or in parallel to suit different hardware constraints.
  arXiv  Detail & Related papers  (2026-03-14T12:56:29Z)
• A Markov Categorical Framework for Language Modeling [9.910562011343009]
  Autoregressive language models achieve remarkable performance, yet a unified theory explaining their internal mechanisms, how training shapes their representations, and enables complex behaviors, remains elusive.<n>We introduce a new analytical framework that models the single-step generation process as a composition of information-processing stages using the language of Markov categories.<n>This work presents a powerful new lens for understanding how information flows through a model and how the training objective shapes its internal geometry.
  arXiv  Detail & Related papers  (2025-07-25T13:14:03Z)
• Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction [57.19302613163439]
  We introduce neural network reprogrammability as a unifying framework for model adaptation.<n>We present a taxonomy that categorizes such information manipulation approaches across four key dimensions.<n>We also analyze remaining technical challenges and ethical considerations.
  arXiv  Detail & Related papers  (2025-06-05T05:42:27Z)
• Self-supervised Latent Space Optimization with Nebula Variational Coding [87.20343320266215]
  This paper proposes a variational inference model which leads to a clustered embedding.<n>We introduce additional variables in the latent space, called textbfnebula anchors, that guide the latent variables to form clusters during training.<n>Since each latent feature can be labeled with the closest anchor, we also propose to apply metric learning in a self-supervised way to make the separation between clusters more explicit.
  arXiv  Detail & Related papers  (2025-06-02T08:13:32Z)
• Analyze Feature Flow to Enhance Interpretation and Steering in Language Models [3.8498574327875947]
  We introduce a new approach to systematically map features discovered by sparse autoencoder across consecutive layers of large language models.<n>By using a data-free cosine similarity technique, we trace how specific features persist, transform, or first appear at each stage.
  arXiv  Detail & Related papers  (2025-02-05T09:39:34Z)
• FedSA: A Unified Representation Learning via Semantic Anchors for Prototype-based Federated Learning [4.244188591221394]
  We propose a novel framework named Federated Learning via Semantic Anchors (FedSA) to decouple the generation of prototypes from local representation learning.<n>FedSA significantly outperforms existing prototype-based FL methods on various classification tasks.
  arXiv  Detail & Related papers  (2025-01-09T16:10:03Z)
• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network)<n>After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference.<n>We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Optimizing against Infeasible Inclusions from Data for Semantic Segmentation through Morphology [58.17907376475596]
  State-of-the-art semantic segmentation models are typically optimized in a data-driven fashion.<n>InSeIn extracts explicit inclusion constraints that govern spatial class relations from the semantic segmentation training set at hand.<n>It then enforces a morphological yet differentiable loss that penalizes violations of these constraints during training to promote prediction feasibility.
  arXiv  Detail & Related papers  (2024-08-26T22:39:08Z)
• Flow Factorized Representation Learning [109.51947536586677]
  We introduce a generative model which specifies a distinct set of latent probability paths that define different input transformations. We show that our model achieves higher likelihoods on standard representation learning benchmarks while simultaneously being closer to approximately equivariant models.
  arXiv  Detail & Related papers  (2023-09-22T20:15:37Z)
• Learning to Generate Training Datasets for Robust Semantic Segmentation [37.9308918593436]
  We propose a novel approach to improve the robustness of semantic segmentation techniques. We design Robusta, a novel conditional generative adversarial network to generate realistic and plausible perturbed images. Our results suggest that this approach could be valuable in safety-critical applications.
  arXiv  Detail & Related papers  (2023-08-01T10:02:26Z)
• Continual Vision-Language Representation Learning with Off-Diagonal Information [112.39419069447902]
  Multi-modal contrastive learning frameworks like CLIP typically require a large amount of image-text samples for training. This paper discusses the feasibility of continual CLIP training using streaming data.
  arXiv  Detail & Related papers  (2023-05-11T08:04:46Z)
• Client-specific Property Inference against Secure Aggregation in Federated Learning [52.8564467292226]
  Federated learning has become a widely used paradigm for collaboratively training a common model among different participants. Many attacks have shown that it is still possible to infer sensitive information such as membership, property, or outright reconstruction of participant data. We show that simple linear models can effectively capture client-specific properties only from the aggregated model updates.
  arXiv  Detail & Related papers  (2023-03-07T14:11:01Z)
• Switchable Representation Learning Framework with Self-compatibility [50.48336074436792]
  We propose a Switchable representation learning Framework with Self-Compatibility (SFSC) SFSC generates a series of compatible sub-models with different capacities through one training process. SFSC achieves state-of-the-art performance on the evaluated datasets.
  arXiv  Detail & Related papers  (2022-06-16T16:46:32Z)
• Federated Self-Supervised Contrastive Learning via Ensemble Similarity Distillation [42.05438626702343]
  This paper investigates the feasibility of learning good representation space with unlabeled client data in a federated scenario. We propose a novel self-supervised contrastive learning framework that supports architecture-agnostic local training and communication-efficient global aggregation.
  arXiv  Detail & Related papers  (2021-09-29T02:13:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.