VetIoT: On Vetting IoT Defenses Enforcing Policies at Runtime

• URL: http://arxiv.org/abs/2308.12417v3
• Date: Tue, 23 Sep 2025 19:50:57 GMT
• Title: VetIoT: On Vetting IoT Defenses Enforcing Policies at Runtime
• Abstract summary: A class of defense solutions dynamically enforces policies that capture the expected behavior of the IoT system.<n>Despite numerous innovations, these solutions are under-vetted.<n>We propose VetIoT -- a highly automated, uniform evaluation platform.
• Score: 3.7924984457839415
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Smart homes, powered by programmable IoT platforms, often face safety and security issues. A class of defense solutions dynamically enforces policies that capture the expected behavior of the IoT system. Despite numerous innovations, these solutions are under-vetted. The primary reason lies in their evaluation approach -- they are self-assessed in isolated virtual testbeds with hand-crafted orchestrated scenarios that require manual interactions using the platform's user-interface (UI). Such non-uniform evaluation setups limit reproducibility and comparative analysis. Closing this gap in the traditional way requires a significant upfront manual effort, causing researchers to turn away from large-scale comparative empirical evaluation. To address this, we propose VetIoT -- a highly automated, uniform evaluation platform -- to vet the defense solutions that hinge on runtime policy enforcement. Given a defense solution, VetIoT readily instantiates a virtual testbed to deploy and evaluate the solution. VetIoT replaces manual UI-based interactions with an automated event simulator and manual inspection of test outcomes with an automated comparator. VetIoT incorporates automated event generators to feed events to the event simulator. We developed a prototype of VetIoT, which successfully reproduced and comparatively assessed four runtime policy enforcement solutions. VetIoT's stress testing and differential testing capabilities make it a promising tool for future research and evaluation.

Related papers

• Context-Aware Generative AI for Automated Telecom Test Script Generation [0.0024792470319550896]
  Existing solutions generate static test suites against a snapshot of the system.<n>As code, configurations, topologies, and key performance indicators (KPIs) evolve, these tests quickly become outdated or misaligned with the live system.<n>This paper presents a context-aware generative AI framework for automated telecom test script generation.
  arXiv  Detail & Related papers  (2026-06-19T06:44:31Z)
• ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving [56.96697737229895]
  Reactive capability is a key property of data-driven behavior world model simulators for autonomous driving simulation systems.<n>We introduce ReactSim-Bench for evaluating the reactive capability of behavior world model simulation in autonomous driving.
  arXiv  Detail & Related papers  (2026-06-12T03:11:06Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• BYOT-CPS: A Hybrid Cyber-Physical Systems Testbed for IoT Security Assessment and Platform Evaluation [0.0]
  This paper presents Build Your Own Cyber-Physical Systems Testbed (BYOT-CPS)<n> BYOT-CPS is a hybrid cyber-physical testbed that connects real IoT devices to virtualised network infrastructure built on GNS3.<n>A prototype deployment integrating smart bulbs, smart plugs, switches, and IP cameras is used to demonstrate hybrid connectivity, testing penetration, a Mirai-style denial-of-service attack, traffic monitoring, and controlled device manipulation.
  arXiv  Detail & Related papers  (2026-05-21T21:49:14Z)
• VP-AutoTest: A Virtual-Physical Fusion Autonomous Driving Testing Platform [47.03629760732478]
  We propose the Virtual-Physical Testing Platform for Autonomous Vehicles (VP-AutoTest) to replicate the diversity of real-world traffic participants.<n>VP-AutoTest incorporates a multidimensional evaluation framework and AI-driven expert systems to conduct comprehensive performance assessment and defect diagnosis.
  arXiv  Detail & Related papers  (2025-12-08T12:43:33Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving [63.875372281596576]
  MetAdv is a novel adversarial testing platform that enables realistic, dynamic, and interactive evaluation.<n>It supports flexible 3D vehicle modeling and seamless transitions between simulated and physical environments.<n>It enables real-time capture of physiological signals and behavioral feedback from drivers.
  arXiv  Detail & Related papers  (2025-08-04T03:07:54Z)
• Test Automation for Interactive Scenarios via Promptable Traffic Simulation [48.240394447516664]
  We introduce an automated method to generate realistic and safety-critical human behaviors for AV planner evaluation in interactive scenarios.<n>We parameterize complex human behaviors using low-dimensional goal positions, which are then fed into a promptable traffic simulator, ProSim.<n>To automate test generation, we introduce a prompt generation module that explores the goal domain and efficiently identifies safety-critical behaviors using Bayesian optimization.
  arXiv  Detail & Related papers  (2025-06-01T22:29:32Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• An Autonomous RL Agent Methodology for Dynamic Web UI Testing in a BDD Framework [0.0]
  This paper introduces an autonomous reinforcement learning (RL) agent integrated within a Behavior-Driven Development (BDD) framework to enhance user interface testing.<n>By leveraging the adaptive decision-making capabilities of RL, the proposed approach dynamically generates and refines test scenarios aligned with specific business expectations and actual user behavior.
  arXiv  Detail & Related papers  (2025-03-11T14:12:50Z)
• Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues [54.81155589931697]
  Collaborative Instance object Navigation (CoIN) is a new task setting where the agent actively resolve uncertainties about the target instance.<n>We propose a novel training-free method, Agent-user Interaction with UncerTainty Awareness (AIUTA)<n>First, upon object detection, a Self-Questioner model initiates a self-dialogue within the agent to obtain a complete and accurate observation description.<n>An Interaction Trigger module determines whether to ask a question to the human, continue or halt navigation.
  arXiv  Detail & Related papers  (2024-12-02T08:16:38Z)
• A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future Direction [5.742965094549775]
  This paper pioneers a strategic roadmap for simulation-based testing of autonomous systems. Our paper discusses the relevant challenges and obstacles of ACPSs, focusing on test automation and quality assurance.
  arXiv  Detail & Related papers  (2024-05-02T07:42:33Z)
• Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems [80.77917437785773]
  Task-oriented dialogue systems ( TDSs) are assessed mainly in an offline setting or through human evaluation. We propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates user's analogical thinking in interactions with systems. We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities.
  arXiv  Detail & Related papers  (2022-04-02T05:11:03Z)
• Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL [14.482670650074885]
  This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique.
  arXiv  Detail & Related papers  (2021-09-16T13:11:53Z)
• Data Driven Testing of Cyber Physical Systems [12.93632948681342]
  We propose an approach to automatically generate fault-revealing test cases for CPS. Data collected from an application managing a smart building have been used to learn models of the environment.
  arXiv  Detail & Related papers  (2021-02-23T04:55:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.