AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming

• URL: http://arxiv.org/abs/2606.24245v2
• Date: Wed, 24 Jun 2026 16:19:48 GMT
• Title: AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming
• Abstract summary: Existing safety approaches face a fundamental tradeoff: hand-crafted rules are interpretable but brittle.<n>We present AutoSpec, a framework that automatically evolves deployed expert-guided safety rules from user safe/unsafe annotations.<n>We evaluate AutoSpec on 291 execution traces spanning code execution and embodied agent domains.
• Score: 21.12573593471532
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents increasingly automate complex tasks by integrating language models with external tools and environments. However, their autonomy poses significant safety risks: agents may execute destructive commands, leak sensitive data, or violate domain constraints. Existing safety approaches face a fundamental tradeoff: hand-crafted rules are interpretable but brittle, with overly conservative rules blocking safe operations (high false positives) while permissive rules miss unsafe behaviors (high false negatives). Neural classifiers lack the interpretability required for safety-critical deployments. We present AutoSpec, a framework that automatically evolves deployed expert-designed safety rules from user safe/unsafe annotations through counterexample-guided inductive synthesis (CEGIS) guided by inductive logic programming (ILP). Starting from the expert rules and a stream of annotated traces, AutoSpec iteratively evaluates rules, mines false-positive and false-negative counterexamples, uses ILP to learn which predicates discriminate them, generates candidate rule edits, and verifies candidates to select the best revision. The key insight is that ILP efficiently identifies predicates that appear frequently in false negatives but rarely in false positives (or vice versa), dramatically pruning the exponential search space of rule edits. This continues until convergence, producing interpretable rules that balance precision and recall. We evaluate AutoSpec on 291 execution traces spanning code execution and embodied agent domains. AutoSpec raises rule F1 to 0.98 and 0.93 across the two domains, achieving up to 94% false positive reduction while maintaining high recall, and converges within 4-5 iterations. The ILP-guided approach achieves up to 4.8x higher F1 than heuristic CEGIS. The learned rules are human-readable, auditable, and generalize to unseen scenarios.

Related papers

• HazardAuditor: From Executable Threats to Safer Computer-Use Agents [50.06351661912873]
  Existing guard models target static prompts and responses.<n>Existing safety platforms produce evaluation verdicts rather than the normalized supervision a guard model needs to learn.<n>We introduce HazardAuditor, an execution-grounded framework that closes both gaps.
  arXiv  Detail & Related papers  (2026-09-14T07:09:33Z)
• CyberLLM: A Multi-Agent LLM Framework for Autonomous Detection and Guarded Response in Automotive Cybersecurity [22.68559728326395]
  CyberLLM is a framework that autonomously detects vulnerabilities and executes remediations under a formal, runtime safety guard.<n>Decision agents aggregates findings, tags them with a human-centric asset taxonomy, and selects a tiered response.<n>A symmetric attack pipeline generates and replays exploits so both sides can be exercised on the same scenarios.
  arXiv  Detail & Related papers  (2026-08-06T23:39:13Z)
• Beyond the Syntax: Do Security Experts Trust LLMs for NIDS Rule Engineering? [70.17374586573033]
  This paper presents a human-centered investigation into Large Language Models (LLMs)-based NIDS rule engineering.<n>Our evaluation reveals a syntax-semantics paradox: although LLMs generate syntactically correct rules, experts find them only partially deployable.<n>While large-scale models consistently produce syntactically valid rules, small models are largely ineffective for IDS rule generation.
  arXiv  Detail & Related papers  (2026-07-07T07:09:56Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• When Rule Violations Are Rare: Chimera Training for Logical Anomaly Detection [56.362776482614976]
  We study anomaly detection in a setting where constraints are given as logical rules over learned visual concepts.<n>We propose a neural rule evaluator that compiles each constraint into a directed acyclic graph and learns feature-aware subtree gates.<n>Across CLEVRER, OpenImages, and VidOR, the resulting evaluator improves rule-level anomaly AUROC over independent-events and same-image semantic-training baselines.
  arXiv  Detail & Related papers  (2026-05-25T02:52:36Z)
• Discovering Agentic Safety Specifications from 1-Bit Danger Signals [6.599344783327054]
  EPO-Safe is a framework where an agent iteratively generates action plans, receives sparse binary danger warnings, and evolves a natural language behavioral specification through reflection.<n>We show that EPO-Safe can perform safety reasoning from a strictly impoverished signal in structured, low-dimensional environments.<n>We show that standard reward-driven reflection actively degrades safety, proving that reflection must be paired with a dedicated safety channel.
  arXiv  Detail & Related papers  (2026-04-25T08:35:36Z)
• Do Agent Rules Shape or Distort? Guardrails Beat Guidance in Coding Agents [9.989306175511238]
  Rules improve performance by 7--14 percentage points, but random rules help as much as expert-curated ones.<n>Individual rules are mostly harmful in isolation yet collectively helpful, with no degradation up to 50 rules.
  arXiv  Detail & Related papers  (2026-04-13T07:10:01Z)
• RulePilot: An LLM-Powered Agent for Security Rule Generation [15.638534363355342]
  Rule-based detection often identifies malicious logs based on the predefined grammar logic.<n>In this paper, we propose RulePilot, which mimics human expertise via LLM-based agent.<n>Using RulePilot, the security analysts do not need to write down the rules following the grammar, instead, they can just provide the annotations.
  arXiv  Detail & Related papers  (2025-11-15T13:59:16Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoning [58.57194301645823]
  Large language models (LLMs) are increasingly integrated into real-world personalized applications.<n>The valuable and often proprietary nature of the knowledge bases used in RAG introduces the risk of unauthorized usage by adversaries.<n>Existing methods that can be generalized as watermarking techniques to protect these knowledge bases typically involve poisoning or backdoor attacks.<n>We propose name for harmless' copyright protection of knowledge bases.
  arXiv  Detail & Related papers  (2025-02-10T09:15:56Z)
• Can LLMs Follow Simple Rules? [28.73820874333199]
  Rule-following Language Evaluation Scenarios (RuLES) is a framework for measuring rule-following ability in Large Language Models. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. We show that almost all current models struggle to follow scenario rules, even on straightforward test cases.
  arXiv  Detail & Related papers  (2023-11-06T08:50:29Z)
• Sample-Efficient Safety Assurances using Conformal Prediction [57.92013073974406]
  Early warning systems can provide alerts when an unsafe situation is imminent. To reliably improve safety, these warning systems should have a provable false negative rate. We present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics.
  arXiv  Detail & Related papers  (2021-09-28T23:00:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.