Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study

• URL: http://arxiv.org/abs/2604.03070v1
• Date: Fri, 03 Apr 2026 14:50:16 GMT
• Title: Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study
• Abstract summary: Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments.<n>We analyze 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection.<n>We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial)
• Score: 51.717224133855886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments, making leakage risks poorly understood. We present the first large-scale empirical study of this problem, analyzing 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection. We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial). We find that (1) leakage is fundamentally cross-modal: 76.3% require joint analysis of code and natural language, while 3.1% arise purely from prompt injection; (2) debug logging is the primary vector, with print and console.log causing 73.5% of leaks due to stdout exposure to LLMs; and (3) leaked credentials are both exploitable (89.6% without privileges) and persistent, as forks retain secrets even after upstream fixes. After disclosure, all malicious skills were removed and 91.6% of hardcoded credentials were fixed. We release our dataset, taxonomy, and detection pipeline to support future research.

Related papers

• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• Understanding and Mitigating Prompt Leaking Attacks in Real-World LLM-Based Applications [57.33208520608409]
  This paper presents a systematic study of prompt leaking in large language model (LLM)-based applications.<n>We find that over 80% of deployments leak system prompts under realistic adversarial queries.<n>We propose AREA, a practical defense that re-anchors the model's attention using an optimizable soft prompt.
  arXiv  Detail & Related papers  (2026-06-17T04:20:00Z)
• MosaicLeaks:Privacy Risks in Querying-in-the-Open for Deep Research Agents [76.60965179634664]
  Deep research agents increasingly combine private local documents with external tools like web retrieval, creating a privacy risk.<n>We introduce MosaicLeaks, a benchmark of 1,001 deep research tasks that chain private enterprise documents and a public web corpus.<n>We find that models across families and sizes frequently leak at all three levels, that zero-shot privacy prompting reduces but does not eliminate leakage.
  arXiv  Detail & Related papers  (2026-05-29T01:44:35Z)
• Behavioral Integrity Verification for AI Agent Skills [9.127363793428119]
  We formalize this as the behavioral integrity verification (BIV) problem.<n>On 49,943 skills from the OpenClaw registry, the deviation taxonomy reveals a pervasive description-implementation gap.<n>On a 906-skill malicious-skill detection benchmark, BIV reaches an F1 of 0.946, outperforming state-of-the-art rule-based and single-pass LLM baselines.
  arXiv  Detail & Related papers  (2026-05-12T08:41:09Z)
• SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills [0.0]
  SkillSieve is a three-layer detection framework that applies progressively deeper analysis only where needed.<n>On a 400-skill labeled benchmark, SkillSieve achieves 0.800 F1, outperforming ClawVet's 0.421, at an average cost of 0.006 per skill.
  arXiv  Detail & Related papers  (2026-04-08T00:58:48Z)
• Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study [47.60135753021306]
  Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines.<n>No ground-truth dataset exists to characterize the resulting threats.<n>We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills.
  arXiv  Detail & Related papers  (2026-02-06T09:52:27Z)
• Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale [26.757365536859453]
  The rise of AI agent frameworks has introduced agent skills, modular packages containing instructions and executable code that dynamically extend agent capabilities.<n>While this architecture enables powerful customization, skills execute with implicit trust and minimal vetting, creating a significant yet uncharacterized attack surface.<n>We conduct the first large-scale empirical security analysis of this emerging ecosystem, collecting 42,447 skills from two major marketplaces.
  arXiv  Detail & Related papers  (2026-01-15T12:31:52Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoning [58.57194301645823]
  Large language models (LLMs) are increasingly integrated into real-world personalized applications.<n>The valuable and often proprietary nature of the knowledge bases used in RAG introduces the risk of unauthorized usage by adversaries.<n>Existing methods that can be generalized as watermarking techniques to protect these knowledge bases typically involve poisoning or backdoor attacks.<n>We propose name for harmless' copyright protection of knowledge bases.
  arXiv  Detail & Related papers  (2025-02-10T09:15:56Z)
• A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish [47.3916421056009]
  Large Language Models (LLMs) are trained on massive web-crawled corpora. LLMs produce leaked information in most cases despite less such data in their training set. Self-detection method showed superior performance compared to existing detection methods.
  arXiv  Detail & Related papers  (2024-03-24T13:21:58Z)
• Boosting Static Resource Leak Detection via LLM-based Resource-Oriented Intention Inference [14.783216988363804]
  Existing static detection techniques rely on mechanical matching of predefined resource acquisition/release APIs and null-checking conditions to find unreleased resources.<n>We propose InferROI, a novel approach that directly infers resource-oriented intentions (acquisition, release, and reachability validation) in code.<n>We evaluate the effectiveness of InferROI in both resource-oriented intention inference and resource leak detection.
  arXiv  Detail & Related papers  (2023-11-08T04:19:28Z)
• Automated Mapping of Vulnerability Advisories onto their Fix Commits in Open Source Repositories [7.629717457706326]
  We present an approach that combines practical experience and machine-learning (ML) An advisory record containing key information about a vulnerability is extracted from an advisory. A subset of candidate fix commits is obtained from the source code repository of the affected project.
  arXiv  Detail & Related papers  (2021-03-24T17:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.