Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis

• URL: http://arxiv.org/abs/2605.00314v1
• Date: Fri, 01 May 2026 00:48:47 GMT
• Title: Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis
• Abstract summary: We present Semia, a static auditor for agent skills.<n>Semia lifts each skill into the Skill Description Language (SDL), a Datalog fact base.<n>We evaluate Semia on 13,728 real-world skills from public marketplaces.
• Score: 11.214847720120192
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: An agent skill is a configuration package that equips an LLM-driven agent with a concrete capability, such as reading email, executing shell commands, or signing blockchain transactions. Each skill is a hybrid artifact-a structured half declares executable interfaces, while a prose half dictates when and how those interfaces fire-and the prose is reinterpreted probabilistically on every invocation. Conventional static analyzers parse the structured half but ignore the prose; LLM-based tools read the prose but cannot reproducibly prove that a tainted input reaches a high-impact sink. We present Semia, a static auditor for agent skills. Semia lifts each skill into the Skill Description Language (SDL), a Datalog fact base that captures LLM-triggered actions, prose-defined conditions, and human-in-the-loop checkpoints. Synthesizing a fact base that is both structurally sound and semantically faithful to the original prose is the central challenge; we address it with Constraint-Guided Representation Synthesis (CGRS), a propose-verify-evaluate loop that refines LLM candidates until convergence. Security properties (e.g., indirect injection, secret leakage, confused deputies, unguarded sinks, etc.) over an agent skill can then be reduced to Datalog reachability queries. We evaluate Semia on 13,728 real-world skills from public marketplaces. Semia renders all of them auditable and finds that more than half carry at least one critical semantic risk. On a stratified sample of 541 expert-labeled skills, Semia achieves 97.7% recall and an F1 of 90.6%, substantially outperforming signature-based scanners and LLM baselines.

Related papers

• ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents [24.540638169054812]
  A single agent hijacked by a malicious third-party skill can cause data exfiltration, privilege escalation, or cascading compromise.<n>We present ClawSentry, an open-source, framework-agnostic security supervision gateway for agent runtimes.
  arXiv  Detail & Related papers  (2026-08-21T13:47:51Z)
• VASO: Formally Verifiable Self-Evolving Skills for Physical AI Agents [57.240036084348354]
  We introduce VASO, a framework for verification-guided self-evolution of robot skill contracts.<n>VASO verifies logically inconsistent skill contracts, then verifies plans induced by the skill against global and local temporal specifications.<n>On Clearpath Jackal and PX4 quadcopter tasks, VASO reaches 97.2% formal-specification compliance using fewer than 100 optimization samples.
  arXiv  Detail & Related papers  (2026-06-03T20:02:35Z)
• Skill-as-Pseudocode: Refactoring Skill Libraries to Pseudocode for LLM Agents [50.78646963535282]
  We propose Skill-as-Pseudocode (SaP), an automatic conversion of markdown skill libraries into typed pseudocode.<n>For each cluster of similar procedural passages drawn from one or more skills, SaP extracts a typed contract and filters it through a four-check deterministic verifier.<n>Promoted contracts are inlined into a rewritten skill skeleton together with restored concrete action templates, giving the agent two complementary signals.
  arXiv  Detail & Related papers  (2026-05-27T04:48:40Z)
• Securing LLM Agents Need Intent-to-Execution Integrity [49.490963596514185]
  We argue that securing LLM agents requires defining an end-to-end correctness property that specifies when an agent's execution faithfully reflects the user's intent.<n>We observe that LLM agents are structurally analogous to compilers, where security violations correspond to mis-executions that do not preserve user intent.<n>We derive four integrity properties that must hold simultaneously: emphTool Integrity, emphInstruction Integrity, emphJudgment Integrity, and emphData Flow Integrity.
  arXiv  Detail & Related papers  (2026-05-16T12:53:31Z)
• Methods for Formal Verification of Agent Skills: Three Layers Toward a Mechanically Checkable Capability-Containment Proof [0.0]
  We give a precise semantics for skill behaviour faithful to how a skill is consumed by an LLM-driven runtime.<n>We present three composable methods that together raise a skill from declared or tested to formal.<n>All three methods plus the bundle producer and re-checker ship as zero-dependency JavaScript modules in the open-source enclawed framework.
  arXiv  Detail & Related papers  (2026-05-09T19:27:38Z)
• Sealing the Audit-Runtime Gap for LLM Skills [22.86837061055991]
  We present SIGIL, the first framework that seals the audit-runtime gap for LLM skills.<n>We evaluate SIGIL on a real-world deployment against 1,023 in-the-wild skills spanning six attack types.
  arXiv  Detail & Related papers  (2026-05-06T14:23:22Z)
• Bian Que: An Agentic Framework with Flexible Skill Arrangement for Online System Operations [19.829321356625428]
  Bian Que is an agentic framework for monitoring, alert response, and root cause analysis.<n>It was deployed on the e-commerce search engine of KuaiShou, the major short-video platform in China.<n>It reduces alert volume by 75%, achieves 80% root-cause analysis accuracy, and cuts mean time to resolution by over 50%.
  arXiv  Detail & Related papers  (2026-04-29T15:35:01Z)
• RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents [0.9821874476902969]
  LLM agents increasingly perform end-to-end ML engineering tasks where success is judged by a single scalar test metric.<n>This creates a structural vulnerability: an agent can increase the reported score by compromising the evaluation pipeline rather than improving the model.<n>We introduce RewardHackingAgents, a workspace-based benchmark that makes two compromise vectors explicit and measurable.
  arXiv  Detail & Related papers  (2026-03-11T22:06:44Z)
• RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis [78.32151470154422]
  We introduce RAVEL, an agentic framework that enables the testers to autonomously plan and execute typical synthesis operations.<n>We present C3EBench, a benchmark comprising 1,258 samples derived from professional human writings.<n>By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability.
  arXiv  Detail & Related papers  (2026-02-28T14:47:34Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning [63.109585527799005]
  GroundingAgent is a visual grounding framework that operates without task-specific fine-tuning.<n>It achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks.<n>It also offers strong interpretability, transparently illustrating each reasoning step.
  arXiv  Detail & Related papers  (2025-11-24T03:11:08Z)
• Distilling LLM Agent into Small Models with Retrieval and Code Tools [65.73762766854192]
  Agent Distillation is a framework for transferring reasoning capability and task-solving behavior from large language models into small language models.<n>Our results show that sLMs as small as 0.5B, 1.5B, 3B parameters can achieve performance competitive with next-tier larger 1.5B, 3B, 7B models.
  arXiv  Detail & Related papers  (2025-05-23T08:20:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.