Which Neurons Detect Malicious Code? A Probing Study of LLM Security Knowledge

• URL: http://arxiv.org/abs/2607.10221v1
• Date: Sat, 11 Jul 2026 09:21:01 GMT
• Title: Which Neurons Detect Malicious Code? A Probing Study of LLM Security Knowledge
• Abstract summary: Large language models (LLMs) have become increasingly capable of understanding and generating source code.<n>We investigated where the malware detection behavior is encoded inside LLMs Feed Forward Network (FFN) neurons.<n>This aims to identify the most important neurons in detecting malicious code.
• Score: 1.6986037642844662
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Background. Large language models (LLMs) have become increasingly capable of understanding and generating source code, leading to their widespread adoption in software engineering tasks such as code completion, repair, and vulnerability detection. However, despite their strong empirical performance, the internal mechanisms through which LLMs recognize malicious or vulnerable code patterns remain poorly understood. Aim. We investigated where the malware detection behavior is encoded inside LLMs Feed Forward Network (FFN) neurons and verified the attribution with causal interventions on the neurons identified. This aims to identify the most important neurons in detecting malicious code. Methods. We applied mechanistic interpretability methods to locate the neurons being responsible for malware-detection behavior in three instruction-tuned LLMs: Llama3.1-8B-Instruct, Mistral-v0.3-7B-Instruct, and Qwen2.5-7B-Instruct. Using 1,500 malicious and 1,500 benign PyPI packages from the PyPI Malregistry, we attribute the behavior to a set of neurons. Results. The experimental results reveal that amplifying facilitating neurons for malware detection while suppressing inhibiting ones can boost accuracy, while the reverse collapses predictions toward a single class, although the magnitude and consistency is heavily model-dependent. We demonstrated that the guardrail detection mechanism varies across models, each represents its malware detection behavior differently within its FFN layers. Conclusions. Probing the neurons associated with security-relevant knowledge helps us gain insights into how LLMs encode malicious programming concepts, identify potentially harmful memorized behaviors, paving the way toward more reliable defense mechanisms, such as neuron-level editing, selective unlearning, and security-aware alignment for code-focused LLMs.

Related papers

• Detecting Contaminated Code-Generation Prompt Batches via Influence Functions [4.185678330063222]
  Large language models (LLMs) are increasingly used for code generation, yet they remain vulnerable to prompts that elicit insecure implementations.<n>We propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts that induce anomalous model behavior.
  arXiv  Detail & Related papers  (2026-08-14T13:37:53Z)
• No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks [61.07406641465909]
  We propose distributed safety alignment, which encodes safety capabilities across multiple computational neurons.<n>We show that DSA substantially improves against white-box neuron-level safety attacks while preserving the model's general language and multimodal utility.
  arXiv  Detail & Related papers  (2026-08-02T17:49:07Z)
• Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection [1.9298849497208217]
  We analyze the internal computations of a neural network to understand its reasoning process.<n>Using Circuit Tracer on Gemma-2-2b, we trace the computational pathways activated when the model classifies 472 C/C++ code samples as vulnerable or safe.
  arXiv  Detail & Related papers  (2026-05-28T13:23:13Z)
• Fuzzing the brain: Automated stress testing for the safety of ML-driven neurostimulation [6.763690463901025]
  Machine learning models are increasingly used to generate electrical stimulation patterns in neuroprosthetic devices such as visual prostheses.<n>We propose a systematic, quantitative approach to detect and characterize unsafe stimulation patterns in ML-driven neurostimulation systems.
  arXiv  Detail & Related papers  (2025-12-05T02:43:22Z)
• Ensembling Large Language Models for Code Vulnerability Detection: An Empirical Evaluation [69.8237598448941]
  This study investigates the potential of ensemble learning to enhance the performance of Large Language Models (LLMs) in source code vulnerability detection.<n>We propose Dynamic Gated Stacking (DGS), a Stacking variant tailored for vulnerability detection.
  arXiv  Detail & Related papers  (2025-09-16T03:48:22Z)
• Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations [2.759846687681801]
  Large language models (LLMs) can sometimes report the strategies they actually use to solve tasks, yet at other times seem unable to recognize those strategies that govern their behavior.<n>This suggests a limited degree of metacognition - the capacity to monitor one's own cognitive processes for subsequent reporting and self-control.<n>We introduce a neuroscience-inspired neurofeedback paradigm that uses in-context learning to quantify metacognitive abilities of LLMs to report and control their activation patterns.
  arXiv  Detail & Related papers  (2025-05-19T22:32:25Z)
• Defending against Indirect Prompt Injection by Instruction Detection [109.30156975159561]
  InstructDetector is a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks.<n>InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark.
  arXiv  Detail & Related papers  (2025-05-08T13:04:45Z)
• NeuRel-Attack: Neuron Relearning for Safety Disalignment in Large Language Models [14.630626774362606]
  Safety alignment in large language models (LLMs) is achieved through fine-tuning mechanisms that regulate neuron activations to suppress harmful content.<n>We propose a novel approach to induce disalignment by identifying and modifying the neurons responsible for safety constraints.
  arXiv  Detail & Related papers  (2025-04-29T05:49:35Z)
• Meta-Representational Predictive Coding: Biomimetic Self-Supervised Learning [51.22185316175418]
  We present a new form of predictive coding that we call meta-representational predictive coding (MPC)<n>MPC sidesteps the need for learning a generative model of sensory input by learning to predict representations of sensory input across parallel streams.
  arXiv  Detail & Related papers  (2025-03-22T22:13:14Z)
• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons [57.07507194465299]
  Large language models (LLMs) excel in various capabilities but pose safety risks such as generating harmful content and misinformation, even after safety alignment.<n>We focus on identifying and analyzing safety neurons within LLMs that are responsible for safety behaviors.<n>We propose inference-time activation contrasting to locate these neurons and dynamic activation patching to evaluate their causal effects on model safety.
  arXiv  Detail & Related papers  (2024-06-20T09:35:22Z)
• Fault-Tolerant Neural Networks from Biological Error Correction Codes [45.82537918529782]
  In the grid cells of the mammalian cortex, analog error correction codes have been observed to protect states against neural spiking noise. Here, we use these biological error correction codes to develop a universal fault-tolerant neural network that achieves reliable computation if the faultiness of each neuron lies below a sharp threshold. The discovery of a phase transition from faulty to fault-tolerant neural computation suggests a mechanism for reliable computation in the cortex.
  arXiv  Detail & Related papers  (2022-02-25T18:55:46Z)
• Towards Efficient Processing and Learning with Spikes: New Approaches for Multi-Spike Learning [59.249322621035056]
  We propose two new multi-spike learning rules which demonstrate better performance over other baselines on various tasks. In the feature detection task, we re-examine the ability of unsupervised STDP with its limitations being presented. Our proposed learning rules can reliably solve the task over a wide range of conditions without specific constraints being applied.
  arXiv  Detail & Related papers  (2020-05-02T06:41:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.