NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation

• URL: http://arxiv.org/abs/2603.12378v1
• Date: Thu, 12 Mar 2026 18:57:20 GMT
• Title: NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
• Abstract summary: Low-Rank Adaptation (LoRA) has become essential for adapting Large Language Models (LLMs) to downstream tasks.<n>Recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference.<n>We propose NeuroLoRA, a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation.<n>NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios.
• Score: 23.036082606498564
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly Low-Rank Adaptation (LoRA), have become essential for adapting Large Language Models (LLMs) to downstream tasks. While the recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference, it relies on a static, magnitude-based routing mechanism that is agnostic to input context. In this paper, we propose NeuroLoRA, a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation -- the dynamic regulation of neuronal excitability based on context. NeuroLoRA retains the computational efficiency of frozen random projections while introducing a lightweight, learnable neuromodulation gate that contextually rescales the projection space prior to expert selection. We further propose a Contrastive Orthogonality Loss to explicitly enforce separation between expert subspaces, enhancing both task decoupling and continual learning capacity. Extensive experiments on MMLU, GSM8K, and ScienceQA demonstrate that NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios, while maintaining comparable parameter efficiency.

Related papers

• Neuron-Guided Fine-Tuning: Unlocking Efficient Alignment Mechanisms for Large Language Models [73.88918843965155]
  Existing Supervised Fine-Tuning paradigms are often plagued by parameter redundancy, inconsistent data quality, and catastrophic forgetting.<n>We propose Neuron-Guided Fine-Tuning (NGFT), a holistic framework that leverages neuron activation patterns as a universal proxy to unify the fine-tuning lifecycle.
  arXiv  Detail & Related papers  (2026-09-05T06:13:24Z)
• Neuro-RIT: Neuron-Guided Instruction Tuning for Robust Retrieval-Augmented Language Model [18.49378483853084]
  Neuro-RIT (Neuron-guided Robust Instruction Tuning) is a novel framework that shifts the paradigm from dense adaptation to precision-driven neuron alignment.<n>Our method explicitly disentangles neurons that are responsible for processing relevant versus irrelevant contexts using attribution-based neuron mining.
  arXiv  Detail & Related papers  (2026-04-02T15:49:50Z)
• Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer [65.72553715508691]
  We show that large vision-language models (LVLMs) lag behind strong text-only large language models (LLMs) on tasks that require multi-step inference and compositional decision-making.<n>We propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference circuitry from LLMs to LVLMs.<n>Our results demonstrate that shared neurons form an interpretable bridge between LLMs and LVLMs, enabling low-cost transfer of inference ability into multimodal models.
  arXiv  Detail & Related papers  (2026-02-22T06:04:05Z)
• NeuroMambaLLM: Dynamic Graph Learning of fMRI Functional Connectivity in Autistic Brains Using Mamba and Language Model Reasoning [0.0]
  We propose NeuroMambaLLM, an end-to-end framework that integrates dynamic latent graph learning and selective state-space temporal modelling with Large Language Models (LLMs)<n>The proposed method learns the functional connectivity dynamically from raw Blood-Oxygen-Level-Dependent (BOLD) time series, replacing fixed correlation graphs with adaptive latent connectivity while suppressing motion-related artifacts and capturing long-range temporal dependencies.<n>This design enables the LLM to perform both diagnostic classification and language-based reasoning, allowing it to analyze dynamic fMRI patterns and generate clinically meaningful textual reports.
  arXiv  Detail & Related papers  (2026-02-14T13:32:59Z)
• Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steering [50.63386303357225]
  We propose AdaRAS, a lightweight test-time framework that improves reasoning reliability by selectively intervening on neuron activations.<n>AdaRAS identifies Reasoning-Critical Neurons (RCNs) via a polarity-aware mean-difference criterion and adaptively steers their activations during inference.<n> Experiments on 10 mathematics and coding benchmarks demonstrate consistent improvements, including over 13% gains on AIME-24 and AIME-25.
  arXiv  Detail & Related papers  (2026-01-27T17:53:01Z)
• Neuronal Group Communication for Efficient Neural representation [85.36421257648294]
  This paper addresses the question of how to build large neural systems that learn efficient, modular, and interpretable representations.<n>We propose Neuronal Group Communication (NGC), a theory-driven framework that reimagines a neural network as a dynamical system of interacting neuronal groups.<n>NGC treats weights as transient interactions between embedding-like neuronal states, with neural computation unfolding through iterative communication among groups of neurons.
  arXiv  Detail & Related papers  (2025-10-19T14:23:35Z)
• Dendritic Resonate-and-Fire Neuron for Effective and Efficient Long Sequence Modeling [66.0841376808143]
  Dendritic Resonate-and-Fire (RF) neurons can efficiently extract frequency from input signals and encode them into spike trains.<n>RF neurons exhibit limited effective memory capacity and a trade-off between energy efficiency and training speed on complex tasks.<n>We propose a Dendritic Resonate-and-Fire (D-RF) model, which explicitly incorporates a multi-dendritic and soma architecture.
  arXiv  Detail & Related papers  (2025-09-21T18:15:45Z)
• Bridging Foundation Models and Efficient Architectures: A Modular Brain Imaging Framework with Local Masking and Pretrained Representation Learning [7.591083752535149]
  We propose a modular framework that integrates principles from foundation models (FM) with efficient, domain-specific architectures.<n>Our framework achieved mean absolute errors (MAEs) of 5.343 for age prediction and 2.940 for fluid intelligence, with Pearson correlation coefficients (PCCs) of 0.928 and 0.887, respectively.<n>This work provides a robust, interpretable alternative to LLM-based approaches for fMRI analysis, offering novel insights into brain aging and cognitive function.
  arXiv  Detail & Related papers  (2025-08-09T08:06:01Z)
• Deep-Unrolling Multidimensional Harmonic Retrieval Algorithms on Neuromorphic Hardware [78.17783007774295]
  This paper explores the potential of conversion-based neuromorphic algorithms for highly accurate and energy-efficient single-snapshot multidimensional harmonic retrieval.<n>A novel method for converting the complex-valued convolutional layers and activations into spiking neural networks (SNNs) is developed.<n>The converted SNNs achieve almost five-fold power efficiency at moderate performance loss compared to the original CNNs.
  arXiv  Detail & Related papers  (2024-12-05T09:41:33Z)
• Advancing Spatio-Temporal Processing in Spiking Neural Networks through Adaptation [6.233189707488025]
  neural networks on neuromorphic hardware promise orders of less power consumption than their non-spiking counterparts.<n>Standard neuron model for spike-based computation on such systems has long been the integrate-and-fire (LIF) neuron.<n>The root of these so-called adaptive LIF neurons is not well understood.
  arXiv  Detail & Related papers  (2024-08-14T12:49:58Z)
• Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model [43.107778640669544]
  Large Language Models (LLMs) are composed of neurons that exhibit various behaviors and roles. Recent studies have revealed that not all neurons are active across different datasets. We introduce Neuron-Level Fine-Tuning (NeFT), a novel approach that refines the granularity of parameter training down to the individual neuron.
  arXiv  Detail & Related papers  (2024-03-18T09:55:01Z)
• The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
  We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron. Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
  arXiv  Detail & Related papers  (2023-06-14T13:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.