Ability Transfer and Recovery via Modularized Parameters Localization

• URL: http://arxiv.org/abs/2601.09398v1
• Date: Wed, 14 Jan 2026 11:42:39 GMT
• Title: Ability Transfer and Recovery via Modularized Parameters Localization
• Abstract summary: Large language models can be continually pre-trained or fine-tuned to improve performance in specific domains, languages, or skills.<n>We propose ACT, which localizes ability-relevant channels via activation differences and selectively transfers only the corresponding parameters.
• Score: 35.06445666199432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models can be continually pre-trained or fine-tuned to improve performance in specific domains, languages, or skills, but this specialization often degrades other capabilities and may cause catastrophic forgetting. We investigate how abilities are distributed within LLM parameters by analyzing module activations under domain- and language-specific inputs for closely related models. Across layers and modules, we find that ability-related activations are highly concentrated in a small set of channels (typically <5\%), and these channels are largely disentangled with good sufficiency and stability. Building on these observations, we propose ACT (Activation-Guided Channel-wise Ability Transfer), which localizes ability-relevant channels via activation differences and selectively transfers only the corresponding parameters, followed by lightweight fine-tuning for compatibility. Experiments on multilingual mathematical and scientific reasoning show that ACT can recover forgotten abilities while preserving retained skills. It can also merge multiple specialized models to integrate several abilities into a single model with minimal interference. Our code and data will be publicly released.

Related papers

• Compressed Sensing for Capability Localization in Large Language Models [58.574245030744834]
  Large language models (LLMs) exhibit a wide range of capabilities, including mathematical reasoning, code generation, and linguistic behaviors.<n>We show that many capabilities are highly localized to small subsets of attention heads within Transformer architectures.
  arXiv  Detail & Related papers  (2026-02-11T16:06:59Z)
• SCALPEL: Selective Capability Ablation via Low-rank Parameter Editing for Large Language Model Interpretability Analysis [5.684482188223278]
  Large language models excel across diverse domains, yet their deployment in high-stakes systems remains limited by incomplete understanding of their internal mechanisms.<n>We present SCALPEL, a framework representing capabilities as low-rank subspaces rather than discrete modules.<n>By training LoRA to reduce distinguishing correct from incorrect answers while preserving general language modeling quality, SCALPEL identifies low-rank representations responsible for particular capabilities while remaining disentangled from others.
  arXiv  Detail & Related papers  (2026-01-12T10:54:18Z)
• Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs [56.76586846269894]
  Multimodal Large Language Models (MLLMs) have achieved success across various domains.<n>Despite its importance, the study of knowledge sharing among domain-specific MLLMs remains largely underexplored.<n>We propose a unified parameter integration framework that enables modular composition of expert capabilities.
  arXiv  Detail & Related papers  (2025-06-30T15:07:41Z)
• The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs [45.08958917457921]
  Large language models (LLMs) still struggle across tasks outside of high-resource languages.<n>In this work, we investigate cross-lingual transfer to lower-resource languages where task-specific post-training data is scarce.
  arXiv  Detail & Related papers  (2025-05-23T20:28:31Z)
• Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities [69.26544016976396]
  We exploit the redundancy within Mixture-of-Experts (MoEs) as a source of additional capacity for learning a new modality.<n>We preserve the original language generation capabilities by applying low-rank adaptation exclusively to the tokens of the new modality.
  arXiv  Detail & Related papers  (2025-03-28T15:21:24Z)
• Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration [104.52015641099828]
  Existing research excels in removing such parameters or merging the outputs of multiple different pretrained models.<n>We propose Compatibility-aware Knowledge Integration (CKI), which consists of Deep Assessment and Deep Splicing.<n>The integrated model can be used directly for inference or for further fine-tuning.
  arXiv  Detail & Related papers  (2025-01-10T01:42:43Z)
• Efficient Transfer Learning for Video-language Foundation Models [13.166348605993292]
  We propose a parameter-efficient Multi-modalpatio Ssupervised-Temporal Adapter (MSTA) to enhance alignment between textual and visual representations.<n>We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-Temporal learning.
  arXiv  Detail & Related papers  (2024-11-18T01:25:58Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• LLM Augmented LLMs: Expanding Capabilities through Composition [56.40953749310957]
  CALM -- Composition to Augment Language Models -- introduces cross-attention between models to compose their representations and enable new capabilities. We illustrate that augmenting PaLM2-S with a smaller model trained on low-resource languages results in an absolute improvement of up to 13% on tasks like translation into English. When PaLM2-S is augmented with a code-specific model, we see a relative improvement of 40% over the base model for code generation and explanation tasks.
  arXiv  Detail & Related papers  (2024-01-04T18:53:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.