From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models

• URL: http://arxiv.org/abs/2604.17941v1
• Date: Mon, 20 Apr 2026 08:21:06 GMT
• Title: From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models
• Abstract summary: HONES is a gradient-free framework for task-aware neuron attribution and steering in vision-language models.<n>We show that HONES outperforms existing methods in identifying task-critical neurons and improves model performance after steering.
• Score: 10.052877942432783
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has increasingly explored neuron-level interpretation in vision-language models (VLMs) to identify neurons critical to final predictions. However, existing neuron analyses generally focus on single tasks, limiting the comparability of neuron importance across tasks. Moreover, ranking strategies tend to score neurons in isolation, overlooking how task-dependent information pathways shape the write-in effects of feed-forward network (FFN) neurons. This oversight can exacerbate neuron polysemanticity in multi-task settings, introducing noise into the identification and intervention of task-critical neurons. In this study, we propose HONES (Head-Oriented Neuron Explanation & Steering), a gradient-free framework for task-aware neuron attribution and steering in multi-task VLMs. HONES ranks FFN neurons by their causal write-in contributions conditioned on task-relevant attention heads, and further modulates salient neurons via lightweight scaling. Experiments on four diverse multimodal tasks and two popular VLMs show that HONES outperforms existing methods in identifying task-critical neurons and improves model performance after steering. Our source code is released at: https://github.com/petergit1/HONES.

Related papers

• Neuron Level Analysis of Large Language Model in Legal Domain Reasoning [27.57551296522159]
  Using neuron attribution scores to rank and suppress influential neurons, we show that suppressing the identified neurons collapses accuracy on the target task.<n>Within the legal domain, the three benchmarks exhibit relatively high neuron overlap and tend to be affected jointly, suggesting of legal components neurons that span jurisdictions.
  arXiv  Detail & Related papers  (2026-06-14T16:06:35Z)
• Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts [57.98275914877019]
  We show that across a wide range of open-weight Transformers, a subset of neurons remains consistently highly activated during inference across tasks of multiple capability dimensions.<n>By probing along the cross-task activation strength, an extremely sparse subset is isolated, whose removal causes a collapse in model behavior.<n>Our analysis reveals that keystone neurons are a stable and intrinsic neuron subset of the model that is largely established during pretraining.
  arXiv  Detail & Related papers  (2026-05-24T03:31:07Z)
• Identifying Good and Bad Neurons for Task-Level Controllable LLMs [43.20582224913806]
  Large Language Models have demonstrated remarkable capabilities on multiple-choice question answering benchmarks.<n>The complex mechanisms underlying their large-scale neurons remain opaque, posing significant challenges for understanding and steering LLMs.<n>We propose NeuronLLM, a novel task-level LLM understanding framework that adopts the biological principle of functional antagonism for LLM neuron identification.
  arXiv  Detail & Related papers  (2026-01-08T03:24:18Z)
• NeuronSeek: On Stability and Expressivity of Task-driven Neurons [19.773883759021764]
  Prototyping task-driven neurons (referred to as NeuronSeek) employs symbolic regression (SR) to discover the optimal neuron formulation.<n>This work replaces symbolic regression with tensor decomposition (TD) to discover optimal neuronal formulations.<n>We establish theoretical guarantees that modifying the aggregation functions with common activation functions can empower a network with a fixed number of parameters to approximate any continuous function with an arbitrarily small error.
  arXiv  Detail & Related papers  (2025-06-01T01:36:27Z)
• Deciphering Functions of Neurons in Vision-Language Models [38.978287253624565]
  This study aims to delve into the internals of vision-language models (VLMs) to interpret the functions of individual neurons.<n>We observe the activations of neurons with respects to the input visual tokens and text tokens, and reveal some interesting findings.<n>We build a framework that automates the explanation of neurons with the assistant of GPT-4o.<n>For visual neurons, we propose an activation simulator to assess the reliability of the explanations for visual neurons.
  arXiv  Detail & Related papers  (2025-02-10T10:00:06Z)
• Artificial Kuramoto Oscillatory Neurons [65.16453738828672]
  It has long been known in both neuroscience and AI that ''binding'' between neurons leads to a form of competitive learning where representations are compressed in order to represent more abstract concepts in deeper layers of the network.<n>We introduce Artificial rethinking together with arbitrary connectivity designs such as fully connected convolutional, or attentive mechanisms.<n>We show that this idea provides performance improvements across a wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, uncertainty, quantification, and reasoning.
  arXiv  Detail & Related papers  (2024-10-17T17:47:54Z)
• Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons [45.04661608619081]
  We detect task-sensitive neurons in large language models (LLMs) via gradient attribution on task-specific data.<n>We find that the overlap of task-specific neurons is strongly associated with generalization and specialization across tasks.<n>We propose a neuron-level continuous fine-tuning method that only fine-tunes the current task-specific neurons during continuous learning.
  arXiv  Detail & Related papers  (2024-07-09T01:27:35Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Compositional Explanations of Neurons [52.71742655312625]
  We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts. We use this procedure to answer several questions on interpretability in models for vision and natural language processing.
  arXiv  Detail & Related papers  (2020-06-24T20:37:05Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.