Understanding NPM Malicious Package Detection: A Benchmark-Driven Empirical Analysis

• URL: http://arxiv.org/abs/2603.27549v1
• Date: Sun, 29 Mar 2026 07:04:31 GMT
• Title: Understanding NPM Malicious Package Detection: A Benchmark-Driven Empirical Analysis
• Abstract summary: NPM ecosystem has become a primary target for software supply chain attacks.<n>We conduct a benchmark-driven empirical analysis of NPM malware detection.<n>We build a dataset of 6,420 malicious and 7,288 benign packages annotated with 11 behavior categories and 8 evasion techniques.
• Score: 10.599261033874884
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: The NPM ecosystem has become a primary target for software supply chain attacks, yet existing detection tools are evaluated in isolation on incompatible datasets, making cross-tool comparison unreliable. We conduct a benchmark-driven empirical analysis of NPM malware detection, building a dataset of 6,420 malicious and 7,288 benign packages annotated with 11 behavior categories and 8 evasion techniques, and evaluating 8 tools across 13 variants. Unlike prior work, we complement quantitative evaluation with source-code inspection of each tool to expose the structural mechanisms behind its performance. Our analysis reveals five key findings. Tool precision-recall positions are structurally determined by how each tool resolves the ambiguity between what code can do and what it intends to do, with GuardDog achieving the best balance at 93.32% F1. A single API call carries no directional intent, but a behavioral chain such as collecting environment variables, serializing, and exfiltrating disambiguates malicious purpose, raising SAP_DT detection from 3.2% to 79.3%. Most malware requires no evasion because the ecosystem lacks mandatory pre-publication scanning. ML degradation stems from concept convergence rather than concept drift: malware became simpler and statistically indistinguishable from benign code in feature space. Tool combination effectiveness is governed by complementarity minus false-positive introduction, not paradigm diversity, with strategic combinations reaching 96.08% accuracy and 95.79% F1. Our benchmark and evaluation framework are publicly available.

Related papers

• The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics [0.0]
  Large language models are increasingly used in blockchain forensic investigations.<n> adversarial deception reduces drain detection by 20.0 percentage points.<n>Models correctly describe the hidden drain mechanism but accept the contract's framing and dismiss it as benign.
  arXiv  Detail & Related papers  (2026-09-12T18:45:24Z)
• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices [48.36933831982682]
  We introduce ADeptS-Bench, a trustworthiness benchmark grounded in the ADEPTS capability framework.<n>The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface.<n>The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous.
  arXiv  Detail & Related papers  (2026-08-25T23:21:57Z)
• MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection [24.37473229751569]
  MaliciousSkillBench is a comprehensive benchmark for malicious Agent Skill detection.<n>We consolidate 13 public sources, 11 of which contribute Core malicious artifacts, and reduce 8,414 raw malicious records to 7,539 normalized-unique identities.<n>We then evaluate three learned text detectors and three off-the-shelf Skill scanners.
  arXiv  Detail & Related papers  (2026-08-20T11:13:00Z)
• Statistical Analysis of Executability and Program Equivalence in Decompilation for IoT Vulnerability Detection [1.9637266103693347]
  We propose a nine-dimensional quality evaluation metric consisting of three categories: structural, behavioral, and semantic similarity.<n>Vulnerabilities often arise in details that are easily lost in this process, such as error-handling flows and boundary checks.<n>This study provides a statistical evaluation foundation for quantifying implementation defects in IoT devices.
  arXiv  Detail & Related papers  (2026-08-07T08:36:35Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software [1.0026496861838445]
  CWE-Trace is a framework for vulnerability detection built from 834 manually curated Linux kernel samples.<n>We evaluate eight vanilla LLMs and 15 LoRA fine-tuned variants across non-targeted detection, targeted detection, and CWE classification.
  arXiv  Detail & Related papers  (2026-06-18T17:19:26Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Security Is Relative: Training-Free Vulnerability Detection via Multi-Agent Behavioral Contract Synthesis [14.657771106188115]
  We propose Phoenix, a training-free multi-agent framework for vulnerability detection.<n>Phoenix decomposes detection into three stages: a Semantic Slicer, a Requirement Reverse Engineer, and a Contract Judge.<n>On PrimeVul Paired, Phoenix achieves F1 = 0.825 and Pair-Correct = 64.4%, surpassing RASM-Vul (F1 = 0.668) and VulTrial (F1 = 0.563)
  arXiv  Detail & Related papers  (2026-04-21T03:02:34Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift [0.0]
  We present a comprehensive analysis using a benchmark of 18 datasets spanning harmful requests, jailbreaks, indirect prompt injections, and extraction attacks.<n>We propose Leave-One-Dataset-Out (LODO) evaluation to measure true out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-02-15T14:21:43Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM [54.38424417079265]
  A Software Bill of Materials (SBOM) is a machine-readable artifact that organizes software information.<n>Following standards, organizations have developed tools for generating and utilizing SBOMs.<n>This paper presents the first large-scale, two-stage empirical analysis of the adherence gap, using our automated evaluation framework, SAP.
  arXiv  Detail & Related papers  (2026-01-09T08:26:05Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluation [94.61617176929384]
  OmniSafeBench-MM is a comprehensive toolbox for multi-modal jailbreak attack-defense evaluation.<n>It integrates 13 representative attack methods, 15 defense strategies, and a diverse dataset spanning 9 major risk domains and 50 fine-grained categories.<n>By unifying data, methodology, and evaluation into an open-source, reproducible platform, OmniSafeBench-MM provides a standardized foundation for future research.
  arXiv  Detail & Related papers  (2025-12-06T22:56:29Z)
• Fine-Grained 1-Day Vulnerability Detection in Binaries via Patch Code Localization [12.73365645156957]
  1-day vulnerabilities in binaries have become a major threat to software security.<n>patch presence test is one of the effective ways to detect the vulnerability.<n>We propose a novel approach named PLocator, which leverages stable values from both the patch code and its context.
  arXiv  Detail & Related papers  (2025-01-29T04:35:37Z)
• Leveraging Large Language Models to Detect npm Malicious Packages [4.479741014073169]
  This study empirically studies the effectiveness of Large Language Models (LLMs) in detecting malicious code.<n>We present SocketAI, a malicious code review workflow to detect malicious code.
  arXiv  Detail & Related papers  (2024-03-18T19:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.