NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs

• URL: http://arxiv.org/abs/2608.08107v1
• Date: Sat, 08 Aug 2026 12:40:43 GMT
• Title: NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
• Abstract summary: Multimodal expansion of large language models compromises language intelligence acquired during pretraining.<n>We propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a framework that allocates neuron-wise update constraints during multimodal instruction tuning.<n>NeuPAT recovers 94.5% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance.
• Score: 15.463007687234743
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.

Related papers

• Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts [57.98275914877019]
  We show that across a wide range of open-weight Transformers, a subset of neurons remains consistently highly activated during inference across tasks of multiple capability dimensions.<n>By probing along the cross-task activation strength, an extremely sparse subset is isolated, whose removal causes a collapse in model behavior.<n>Our analysis reveals that keystone neurons are a stable and intrinsic neuron subset of the model that is largely established during pretraining.
  arXiv  Detail & Related papers  (2026-05-24T03:31:07Z)
• MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation [19.679466742536892]
  Multimodal large language models (MLLMs) have shown impressive capabilities, yet they often struggle to effectively capture the fine-grained textual information within images.<n>Existing methods, primarily relying on instruction fine-tuning, risk parameter redundancy of pre-trained knowledge, hindering generalization performance.<n>We introduce modality neuron-aware fine-tuning (MNAFT), a novel approach that takes advantage of the specialized roles of individual neurons within MLLMs for enhanced image translation.
  arXiv  Detail & Related papers  (2026-04-18T09:54:20Z)
• Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer [65.72553715508691]
  We show that large vision-language models (LVLMs) lag behind strong text-only large language models (LLMs) on tasks that require multi-step inference and compositional decision-making.<n>We propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference circuitry from LLMs to LVLMs.<n>Our results demonstrate that shared neurons form an interpretable bridge between LLMs and LVLMs, enabling low-cost transfer of inference ability into multimodal models.
  arXiv  Detail & Related papers  (2026-02-22T06:04:05Z)
• Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models [11.719190735841407]
  Large language models exhibit uneven performance across languages.<n>We present a framework for enhancing monolingual capabilities of LLMs in underrepresented languages.<n>Our approach identifies language-specific neurons using Language Activation Probability Entropy and fine-tunes only the weights associated with these neurons.
  arXiv  Detail & Related papers  (2025-10-15T14:14:49Z)
• Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models [60.39744129890118]
  Large vision-language models (LVLMs) have demonstrated exceptional capabilities in understanding visual information with human languages.<n>In this work, we identify a salient correlation between the multilingual understanding ability of LVLMs and language-specific neuron activations in shallow layers.<n>We introduce PLAST, a training recipe that achieves efficient multilingual enhancement for LVLMs by Precise LAnguage-Specific layers fine-Tuning.
  arXiv  Detail & Related papers  (2025-08-25T18:15:25Z)
• Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation [9.518772041855923]
  We analyze language-specific neurons in Llama-3.1-8B, Mistral-Nemo-12B, and Aya-Expanse-8B & 32B across 21 typologically diverse languages.<n>We show that these neurons cluster in deeper layers, with non-Latin scripts showing greater specialization.<n>We steer models to deactivate unwanted languages and activate desired ones, outperforming simpler replacement approaches.
  arXiv  Detail & Related papers  (2025-07-30T12:23:39Z)
• The Emergence of Abstract Thought in Large Language Models Beyond Any Language [95.50197866832772]
  Large language models (LLMs) function effectively across a diverse range of languages.<n>Preliminary studies observe that the hidden activations of LLMs often resemble English, even when responding to non-English prompts.<n>Recent results show strong multilingual performance, even surpassing English performance on specific tasks in other languages.
  arXiv  Detail & Related papers  (2025-06-11T16:00:54Z)
• How does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective [64.79894853375478]
  We propose a new finer-grained neuron identification algorithm, which detects language neurons(including language-specific neurons and language-related neurons) and language-agnostic neurons.<n>Based on the distributional characteristics of different types of neurons, we divide the LLMs' internal process for multilingual inference into four parts.<n>We systematically analyze the models before and after alignment with a focus on different types of neurons.
  arXiv  Detail & Related papers  (2025-05-27T17:59:52Z)
• Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs [19.472889262384818]
  We propose a training-free parameter fusion framework that locates important parameters and then selectively merges them.<n>We introduce Neuron-Fusion, a neuron-level strategy that preserves the influence of neurons with large parameter shifts.<n>Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods.
  arXiv  Detail & Related papers  (2025-05-22T14:04:43Z)
• Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation [7.376832526909754]
  Training a unified multilingual model promotes knowledge transfer but inevitably introduces negative interference. We propose Neuron, an approach that identifies specialized neurons to modularize feed-forward layers and then continuously updates them through sparse networks.
  arXiv  Detail & Related papers  (2024-04-17T09:33:19Z)
• Revealing the Parallel Multilingual Learning within Large Language Models [50.098518799536144]
  In this study, we reveal an in-context learning capability of multilingual large language models (LLMs)<n>By translating the input to several languages, we provide Parallel Input in Multiple Languages (PiM) to LLMs, which significantly enhances their comprehension abilities.
  arXiv  Detail & Related papers  (2024-03-14T03:33:46Z)
• How do Large Language Models Handle Multilingualism? [81.15060972112563]
  This study explores how large language models (LLMs) handle multilingualism. LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures.
  arXiv  Detail & Related papers  (2024-02-29T02:55:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.