Localized Adaptation Reveals Distinct Learning Signatures in Transformers

• URL: http://arxiv.org/abs/2607.25663v1
• Date: Tue, 28 Jul 2026 12:49:55 GMT
• Title: Localized Adaptation Reveals Distinct Learning Signatures in Transformers
• Abstract summary: Transformer adaptation is typically distributed across model depth, even when the intended change is narrow.<n>We investigate how adaptation site shapes what a model learns, how well that learning generalizes, and how selectively it is applied.
• Score: 1.4824891788575416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer adaptation is typically distributed across model depth, even when the intended change is narrow. We investigate how adaptation site shapes what a model learns, how well that learning generalizes, and how selectively it is applied. We introduce a controlled benchmark spanning five objectives (lexical binding, factual association, behavioral policy learning, causal mapping, and procedural reasoning) and define each objective's "adaptation geometry" as its profile of acquisition, transfer, and boundedness under full-stack and early-, middle-, or late-layer LoRA. The objectives exhibit distinct geometries. Lexical binding favors early-layer adaptation for acquisition and boundedness but requires broader updates for transfer; factual association favors later layers among localized adapters; behavioral learning separates late-layer action acquisition from middle-layer policy gating; and causal and procedural transfer benefit most from middle- or full-stack adaptation. These patterns largely persist under parameter-matched controls, and most corresponding directional contrasts replicate across five model families. These findings establish adaptation site as a key design variable for controlling what models learn, generalize, and leave unchanged.

Related papers

• Loss Smoothing for Stable Adaptation Under Distribution Shift [33.43907011755832]
  Loss smoothing is a simple approach that interpolates between the source and target training objectives at the start of adaptation.<n>Across controlled supervised shifts, pretrained vision adaptation, offline-to-online and online reinforcement learning, and language model fine-tuning, we find that loss smoothing consistently improves performance.
  arXiv  Detail & Related papers  (2026-07-01T08:47:48Z)
• From Weights to Activations: Is Steering the Next Frontier of Adaptation? [24.89959346599616]
  Post-training adaptation of language models is commonly achieved through parameter updates or input-based methods.<n>We argue that steering should be regarded as a form of model adaptation.
  arXiv  Detail & Related papers  (2026-04-15T17:06:25Z)
• $\boldsymbolλ$-Orthogonality Regularization for Compatible Representation Learning [48.264642951728085]
  Retrieval systems rely on representations learned by increasingly powerful models.<n>Due to the high training cost and inconsistencies in learned representations, there is significant interest in facilitating communication between representations.
  arXiv  Detail & Related papers  (2025-09-20T12:35:07Z)
• GeoAda: Efficiently Finetune Geometric Diffusion Models with Equivariant Adapters [61.51810815162003]
  We propose an SE(3)-equivariant adapter framework ( GeoAda) that enables flexible and parameter-efficient fine-tuning for controlled generative tasks.<n>GeoAda preserves the model's geometric consistency while mitigating overfitting and catastrophic forgetting.<n>We demonstrate the wide applicability of GeoAda across diverse geometric control types, including frame control, global control, subgraph control, and a broad range of application domains.
  arXiv  Detail & Related papers  (2025-07-02T18:44:03Z)
• Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction [57.19302613163439]
  We introduce neural network reprogrammability as a unifying framework for model adaptation.<n>We present a taxonomy that categorizes such information manipulation approaches across four key dimensions.<n>We also analyze remaining technical challenges and ethical considerations.
  arXiv  Detail & Related papers  (2025-06-05T05:42:27Z)
• Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning [21.19820308728003]
  Continual learning (CL) aims to continually accumulate knowledge from a non-stationary data stream without catastrophic forgetting of learned knowledge.<n>Existing PTM-based CL methods use restricted adaptation on a fixed set of these modules to avoid forgetting.<n>We propose Self-Expansion of pre-trained models with Modularized Adaptation (SEMA), a novel approach to enhance the control of stability-plasticity balance in PTM-based CL.
  arXiv  Detail & Related papers  (2024-03-27T17:59:21Z)
• Layerwise complexity-matched learning yields an improved model of cortical area V2 [12.861402235256207]
  Deep neural networks trained end-to-end for object recognition approach human capabilities. We develop a self-supervised training methodology that operates independently on successive layers. We show that our model is better aligned with selectivity properties and neural activity in primate area V2.
  arXiv  Detail & Related papers  (2023-12-18T18:37:02Z)
• Transformer-Based Source-Free Domain Adaptation [134.67078085569017]
  We study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. We propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA.
  arXiv  Detail & Related papers  (2021-05-28T23:06:26Z)
• Self-supervised Augmentation Consistency for Adapting Semantic Segmentation [56.91850268635183]
  We propose an approach to domain adaptation for semantic segmentation that is both practical and highly accurate. We employ standard data augmentation techniques $-$ photometric noise, flipping and scaling $-$ and ensure consistency of the semantic predictions. We achieve significant improvements of the state-of-the-art segmentation accuracy after adaptation, consistent both across different choices of the backbone architecture and adaptation scenarios.
  arXiv  Detail & Related papers  (2021-04-30T21:32:40Z)
• Epigenetic evolution of deep convolutional models [81.21462458089142]
  We build upon a previously proposed neuroevolution framework to evolve deep convolutional models. We propose a convolutional layer layout which allows kernels of different shapes and sizes to coexist within the same layer. The proposed layout enables the size and shape of individual kernels within a convolutional layer to be evolved with a corresponding new mutation operator.
  arXiv  Detail & Related papers  (2021-04-12T12:45:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.