Correct-by-Construction Runtime Enforcement in AI -- A Survey

• URL: http://arxiv.org/abs/2208.14426v1
• Date: Tue, 30 Aug 2022 17:45:38 GMT
• Title: Correct-by-Construction Runtime Enforcement in AI -- A Survey
• Authors: Bettina K\"onighofer, Roderick Bloem, R\"udiger Ehlers, Christian Pek
• Abstract summary: Enforcement refers to the theories, techniques, and tools for enforcing correct behavior with respect to a formal specification of systems at runtime. We discuss how safety is traditionally handled in the field of AI and how more formal guarantees on the safety of a self-learning agent can be given by integrating a runtime enforcer.
• Score: 3.509295509987626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Runtime enforcement refers to the theories, techniques, and tools for enforcing correct behavior with respect to a formal specification of systems at runtime. In this paper, we are interested in techniques for constructing runtime enforcers for the concrete application domain of enforcing safety in AI. We discuss how safety is traditionally handled in the field of AI and how more formal guarantees on the safety of a self-learning agent can be given by integrating a runtime enforcer. We survey a selection of work on such enforcers, where we distinguish between approaches for discrete and continuous action spaces. The purpose of this paper is to foster a better understanding of advantages and limitations of different enforcement techniques, focusing on the specific challenges that arise due to their application in AI. Finally, we present some open challenges and avenues for future work.

Related papers

• AgentOrca: A Dual-System Framework to Evaluate Language Agents on Operational Routine and Constraint Adherence [54.317522790545304]
  We present AgentOrca, a dual-system framework for evaluating language agents' compliance with operational constraints and routines. Our framework encodes action constraints and routines through both natural language prompts for agents and corresponding executable code serving as ground truth for automated verification. Our findings reveal notable performance gaps among state-of-the-art models, with large reasoning models like o1 demonstrating superior compliance while others show significantly lower performance.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Position: A taxonomy for reporting and describing AI security incidents [57.98317583163334]
  We argue that specific are required to describe and report security incidents of AI systems. Existing frameworks for either non-AI security or generic AI safety incident reporting are insufficient to capture the specific properties of AI security.
  arXiv  Detail & Related papers  (2024-12-19T13:50:26Z)
• AuditNet: A Conversational AI-based Security Assistant [DEMO] [10.941722434218262]
  We propose a versatile conversational AI assistant framework designed to facilitate compliance checking on the go. Our framework automates the review, indexing, and retrieval of relevant, context-aware information. This AI assistant not only reduces the manual effort involved in compliance checks but also enhances accuracy and efficiency.
  arXiv  Detail & Related papers  (2024-07-19T08:33:07Z)
• Operationalizing the Blueprint for an AI Bill of Rights: Recommendations for Practitioners, Researchers, and Policy Makers [20.16404495546234]
  Several regulatory frameworks have been introduced by different countries worldwide. Many of these frameworks emphasize the need for auditing and improving the trustworthiness of AI tools. Although these regulatory frameworks highlight the necessity of enforcement, practitioners often lack detailed guidance on implementing them. We provide easy-to-understand summaries of state-of-the-art literature and highlight various gaps that exist between regulatory guidelines and existing AI research.
  arXiv  Detail & Related papers  (2024-07-11T17:28:07Z)
• Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems [88.80306881112313]
  We will introduce and define a family of approaches to AI safety, which we will refer to as guaranteed safe (GS) AI. The core feature of these approaches is that they aim to produce AI systems which are equipped with high-assurance quantitative safety guarantees. We outline a number of approaches for creating each of these three core components, describe the main technical challenges, and suggest a number of potential solutions to them.
  arXiv  Detail & Related papers  (2024-05-10T17:38:32Z)
• Inherent Diverse Redundant Safety Mechanisms for AI-based Software Elements in Automotive Applications [1.6495054381576084]
  This paper explores the role and challenges of Artificial Intelligence (AI) algorithms in autonomous driving systems. A primary concern relates to the ability (and necessity) of AI models to generalize beyond their initial training data. This paper investigates the risk associated with overconfident AI models in safety-critical applications like autonomous driving.
  arXiv  Detail & Related papers  (2024-02-13T04:15:26Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)
• A Case for AI Safety via Law [0.0]
  How to make artificial intelligence (AI) systems safe and aligned with human values is an open research question. Proposed solutions tend toward relying on human intervention in uncertain situations. This paper makes a case that effective legal systems are the best way to address AI safety.
  arXiv  Detail & Related papers  (2023-07-31T19:55:27Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Constrained Reinforcement Learning for Robotics via Scenario-Based Programming [64.07167316957533]
  It is crucial to optimize the performance of DRL-based agents while providing guarantees about their behavior. This paper presents a novel technique for incorporating domain-expert knowledge into a constrained DRL training loop. Our experiments demonstrate that using our approach to leverage expert knowledge dramatically improves the safety and the performance of the agent.
  arXiv  Detail & Related papers  (2022-06-20T07:19:38Z)
• Compliance Challenges in Forensic Image Analysis Under the Artificial Intelligence Act [8.890638003061605]
  We review why the use of machine learning in forensic image analysis is classified as high-risk. Under the draft AI act, high-risk AI systems for use in law enforcement are permitted but subject to compliance with mandatory requirements.
  arXiv  Detail & Related papers  (2022-03-01T14:03:23Z)
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety [54.478842696269304]
  The use of deep neural networks (DNNs) in safety-critical applications is challenging due to numerous model-inherent shortcomings. In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged. Our paper addresses both machine learning experts and safety engineers.
  arXiv  Detail & Related papers  (2021-04-29T09:54:54Z)
• Multi-Agent Reinforcement Learning with Temporal Logic Specifications [65.79056365594654]
  We study the problem of learning to satisfy temporal logic specifications with a group of agents in an unknown environment. We develop the first multi-agent reinforcement learning technique for temporal logic specifications. We provide correctness and convergence guarantees for our main algorithm.
  arXiv  Detail & Related papers  (2021-02-01T01:13:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.