SkillScanner: Detecting Policy-Violating Voice Applications Through Static Analysis at the Development Phase

• URL: http://arxiv.org/abs/2309.05867v1
• Date: Mon, 11 Sep 2023 23:22:34 GMT
• Title: SkillScanner: Detecting Policy-Violating Voice Applications Through Static Analysis at the Development Phase
• Abstract summary: Amazon Alexa has implemented a set of policy requirements to be adhered to by third-party skill developers. Recent works reveal the prevalence of policy-violating skills in the current skills store.
• Score: 24.084878589421113
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Amazon Alexa marketplace is the largest Voice Personal Assistant (VPA) platform with over 100,000 voice applications (i.e., skills) published to the skills store. In an effort to maintain the quality and trustworthiness of voice-apps, Amazon Alexa has implemented a set of policy requirements to be adhered to by third-party skill developers. However, recent works reveal the prevalence of policy-violating skills in the current skills store. To understand the causes of policy violations in skills, we first conduct a user study with 34 third-party skill developers focusing on whether they are aware of the various policy requirements defined by the Amazon Alexa platform. Our user study results show that there is a notable gap between VPA's policy requirements and skill developers' practices. As a result, it is inevitable that policy-violating skills will be published. To prevent the inflow of new policy-breaking skills to the skills store from the source, it is critical to identify potential policy violations at the development phase. In this work, we design and develop SkillScanner, an efficient static code analysis tool to facilitate third-party developers to detect policy violations early in the skill development lifecycle. To evaluate the performance of SkillScanner, we conducted an empirical study on 2,451 open source skills collected from GitHub. SkillScanner effectively identified 1,328 different policy violations from 786 skills. Our results suggest that 32% of these policy violations are introduced through code duplication (i.e., code copy and paste). In particular, we found that 42 skill code examples from potential Alexa's official accounts (e.g., "alexa" and "alexa-samples" on GitHub) contain policy violations, which lead to 81 policy violations in other skills due to the copy-pasted code snippets from these Alexa's code examples.

Related papers

• A Finger on the Scale: Covert Policy Steering through Agentic Skills [42.71897827897363]
  Reusable agent skills extend large language model (LLM) agents with task procedures, tool-use guidance, and output constraints.<n>We formalize Skill Policy Integrity, which requires a Skill-induced policy to remain aligned with its declared functionality and the user-authorized objective.<n>We present SkillShift, a constrained black-box framework for covert policy steering without explicit target command injection or task hijacking.
  arXiv  Detail & Related papers  (2026-09-02T13:15:14Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills [24.371534406647978]
  MalSkillBench is the first runtime-verified benchmark of malicious agent skills.<n>We show that code injection reaches 94.5% verification yield but prompt injection only 75.8%.<n>We release the dataset, pipeline, baselines, and results.
  arXiv  Detail & Related papers  (2026-06-05T10:43:19Z)
• Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
  We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
  arXiv  Detail & Related papers  (2026-05-12T02:11:54Z)
• Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study [51.717224133855886]
  Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments.<n>We analyze 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection.<n>We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial)
  arXiv  Detail & Related papers  (2026-04-03T14:50:16Z)
• Malicious Or Not: Adding Repository Context to Agent Skill Classification [3.773312847224526]
  We present the largest empirical security analysis of the AI agent skill ecosystem.<n>We collect 238,180 unique skills from three major distribution platforms and GitHub.<n>Our analysis also uncovers several, by now undocumented real-world attack vectors.
  arXiv  Detail & Related papers  (2026-03-17T14:27:35Z)
• SoK: Agentic Skills -- Beyond Tool Use in LLM Agents [6.356997609995175]
  Agentic systems increasingly rely on reusable procedural capabilities, textita.k.a., agentic skills, to execute long-horizon reliably.<n>This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update)<n>We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution.
  arXiv  Detail & Related papers  (2026-02-24T13:11:38Z)
• Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study [47.60135753021306]
  Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines.<n>No ground-truth dataset exists to characterize the resulting threats.<n>We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills.
  arXiv  Detail & Related papers  (2026-02-06T09:52:27Z)
• Understanding Code Understandability Improvements in Code Reviews [79.16476505761582]
  We analyzed 2,401 code review comments from Java open-source projects on GitHub. 83.9% of suggestions for improvement were accepted and integrated, with fewer than 1% later reverted.
  arXiv  Detail & Related papers  (2024-10-29T12:21:23Z)
• SAFE-GIL: SAFEty Guided Imitation Learning for Robotic Systems [15.782203322922017]
  We propose SAFE-GIL, a design-time method to learn safety-aware behavior cloning policies. We inject adversarial disturbance in the system during data collection to guide the expert towards safety-critical states. This disturbance injection simulates potential policy errors that the system might encounter during the test time.
  arXiv  Detail & Related papers  (2024-04-08T07:25:25Z)
• Residual Q-Learning: Offline and Online Policy Customization without Value [53.47311900133564]
  Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. We formulate a new problem setting called policy customization. We propose a novel framework, Residual Q-learning, which can solve the formulated MDP by leveraging the prior policy.
  arXiv  Detail & Related papers  (2023-06-15T22:01:19Z)
• Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning [48.75878234995544]
  We propose the Skill-Critic algorithm to fine-tune the low-level policy in conjunction with high-level skill selection. We validate Skill-Critic in multiple sparse-reward environments, including a new sparse-reward autonomous racing task in Gran Turismo Sport.
  arXiv  Detail & Related papers  (2023-06-14T09:24:32Z)
• SkillFence: A Systems Approach to Practically Mitigating Voice-Based Confusion Attacks [9.203566746598439]
  Recent work has shown that commercial systems like Amazon Alexa and Google Home are vulnerable to voice-based confusion attacks. We propose a systems-oriented defense against this class of attacks and demonstrate its functionality for Amazon Alexa. We build SkilIFence, a browser extension that existing voice assistant users can install to ensure that only legitimate skills run in response to their commands.
  arXiv  Detail & Related papers  (2022-12-16T22:22:04Z)
• Choreographer: Learning and Adapting Skills in Imagination [60.09911483010824]
  We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination. Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model. Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy.
  arXiv  Detail & Related papers  (2022-11-23T23:31:14Z)
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals [89.67840168694259]
  We propose to incorporate imagined subgoals into policy learning to facilitate learning of complex tasks. Imagined subgoals are predicted by a separate high-level policy, which is trained simultaneously with the policy and its critic. We evaluate our approach on complex robotic navigation and manipulation tasks and show that it outperforms existing methods by a large margin.
  arXiv  Detail & Related papers  (2021-07-01T15:30:59Z)
• Automated Mapping of Vulnerability Advisories onto their Fix Commits in Open Source Repositories [7.629717457706326]
  We present an approach that combines practical experience and machine-learning (ML) An advisory record containing key information about a vulnerability is extracted from an advisory. A subset of candidate fix commits is obtained from the source code repository of the affected project.
  arXiv  Detail & Related papers  (2021-03-24T17:50:35Z)
• Preventing Imitation Learning with Adversarial Policy Ensembles [79.81807680370677]
  Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. How can we protect against external observers cloning our proprietary policies? We introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies.
  arXiv  Detail & Related papers  (2020-01-31T01:57:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.