FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning

• URL: http://arxiv.org/abs/2603.01135v1
• Date: Sun, 01 Mar 2026 14:33:08 GMT
• Title: FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning
• Abstract summary: Foundations for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks.<n>Existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs.<n>We propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning.
• Score: 6.263790020235786
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN embeddings with the LLM and then jointly fine-tunes the entire model to capture high-level semantic information. Experiments on a large-scale, multi-site FCN database show that FCN-LLM achieves strong zero-shot generalization on unseen datasets, outperforming conventional supervised and foundation models. This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.

Related papers

• Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models [14.282205541707699]
  Spoken Language Understanding (SLU) is the core component of task-oriented dialogue systems.<n>This work proposes Spoken Language Function Calling (SFC) to evolve beyond traditional closed-set SLU.
  arXiv  Detail & Related papers  (2026-08-05T17:50:31Z)
• NeuroMAS: Multi-Agent Systems as Neural Networks with Joint Reinforcement Learning [7.167256048138654]
  We propose NeuroMAS as a trainable and scalable neural-network-like architecture for multi-agent language systems.<n>We show that NeuroMAS improves significantly over both inference-time and trained multi-agent baselines.<n>Larger systems can be challenging to train from scratch, but become feasible when grown progressively from smaller trained systems.
  arXiv  Detail & Related papers  (2026-05-16T02:11:34Z)
• BLEG: LLM Functions as Powerful fMRI Graph-Enhancer for Brain Network Analysis [26.180995145027183]
  Graph Neural Networks (GNNs) have been widely used in diverse brain network analysis tasks based on preprocessed functional magnetic resonance imaging (fMRI) data.<n>However, their performances are constrained due to high feature sparsity and inherent limitations of domain knowledge within uni-modal neurographs.<n>Large language models (LLMs) have demonstrated powerful representation capabilities.
  arXiv  Detail & Related papers  (2026-04-01T06:53:13Z)
• Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer [65.72553715508691]
  We show that large vision-language models (LVLMs) lag behind strong text-only large language models (LLMs) on tasks that require multi-step inference and compositional decision-making.<n>We propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference circuitry from LLMs to LVLMs.<n>Our results demonstrate that shared neurons form an interpretable bridge between LLMs and LVLMs, enabling low-cost transfer of inference ability into multimodal models.
  arXiv  Detail & Related papers  (2026-02-22T06:04:05Z)
• Pruning Large Language Models by Identifying and Preserving Functional Networks [41.601762545495255]
  Structured pruning is a technique for compressing large language models (LLMs) to reduce GPU memory consumption and accelerate inference speed.<n>Most structured pruning methods overlook the interaction and collaboration among artificial neurons that are crucial for the functionalities of LLMs.<n>Inspired by the inherent similarities between artificial neural networks and functional neural networks in the human brain, we propose to prune LLMs by identifying and preserving functional networks.
  arXiv  Detail & Related papers  (2025-08-07T10:27:01Z)
• Probing Neural Topology of Large Language Models [12.298921317333452]
  We introduce graph probing, a method for uncovering the functional connectivity of large language models.<n>By probing models across diverse LLM families and scales, we discover a universal predictability of next-token prediction performance.<n>Strikingly, probing on topology outperforms probing on activation by up to 130.4%.
  arXiv  Detail & Related papers  (2025-06-01T14:57:03Z)
• Brain-Inspired Exploration of Functional Networks and Key Neurons in Large Language Models [53.91412558475662]
  We use methods similar to those in the field of functional neuroimaging analysis to locate and identify functional networks in large language models (LLMs)<n> Experimental results show that, similar to the human brain, LLMs contain functional networks that frequently recur during operation.<n>Masking key functional networks significantly impairs the model's performance, while retaining just a subset is adequate to maintain effective operation.
  arXiv  Detail & Related papers  (2025-02-13T04:42:39Z)
• Concept Bottleneck Large Language Models [30.877541733738536]
  CB-LLMs is a framework for building inherently interpretable Large Language Models.<n>We build CB-LLMs for two essential NLP tasks: text classification and text generation.<n> embedded interpretability empowers users to transparently identify harmful content, steer model behavior, and unlearn undesired concepts.
  arXiv  Detail & Related papers  (2024-12-11T00:04:10Z)
• Neural Topic Modeling with Large Language Models in the Loop [12.142323482188056]
  We propose a novel framework that integrates Large Language Models (LLMs) with Neural Topic Models (NTMs)<n>With the flexibility of being integrated into many existing NTMs, the proposed approach enhances the interpretability of topics while preserving the efficiency of NTMs in learning topics and document representations.
  arXiv  Detail & Related papers  (2024-11-13T11:31:02Z)
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning [54.033346088090674]
  We introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability.<n>To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT.<n>This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• NVLM: Open Frontier-Class Multimodal LLMs [64.00053046838225]
  We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks. We propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. We develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:59:06Z)
• All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks [51.19110891434727]
  Large Language Models (LLMs) with pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data. E-LLaGNN is a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph.
  arXiv  Detail & Related papers  (2024-07-20T22:09:42Z)
• Graph Neural Prompting with Large Language Models [32.97391910476073]
  Graph Neural Prompting (GNP) is a novel plug-and-play method to assist pre-trained language models in learning beneficial knowledge from knowledge graphs. Extensive experiments on multiple datasets demonstrate the superiority of GNP on both commonsense and biomedical reasoning tasks.
  arXiv  Detail & Related papers  (2023-09-27T06:33:29Z)
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding [97.85957811603251]
  We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks. A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm.
  arXiv  Detail & Related papers  (2020-02-19T03:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.