First do no harm: counterfactual objective functions for safe & ethical AI

• URL: http://arxiv.org/abs/2204.12993v1
• Date: Wed, 27 Apr 2022 15:03:43 GMT
• Title: First do no harm: counterfactual objective functions for safe & ethical AI
• Abstract summary: We develop the first statistical definition of harm and a framework for factoring harm into algorithmic decisions. Our results show that counterfactual reasoning is a key ingredient for safe and ethical AI.
• Score: 0.03683202928838612
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions. In this paper we develop the first statistical definition of harm and a framework for factoring harm into algorithmic decisions. We argue that harm is fundamentally a counterfactual quantity, and show that standard machine learning algorithms are guaranteed to pursue harmful policies in certain environments. To resolve this, we derive a family of counterfactual objective functions that robustly mitigate for harm. We demonstrate our approach with a statistical model for identifying optimal drug doses. While identifying optimal doses using the causal treatment effect results in harmful treatment decisions, our counterfactual algorithm identifies doses that are far less harmful without sacrificing efficacy. Our results show that counterfactual reasoning is a key ingredient for safe and ethical AI.

Related papers

• Counterfactually Safe Reinforcement Learning [25.8534002955263]
  Reinforcement learning algorithms are generally designed to maximize the expected return across a population.<n>We first formalize the notion of individual harm from a counterfactual perspective and define harm as the event in which a chosen action results in a strictly worse outcome than a baseline alternative.<n>We then propose a general two-stage procedure for learning policies that maximize the expected return while accounting for individual harm.
  arXiv  Detail & Related papers  (2026-05-24T14:54:06Z)
• SafeMath: Inference-time Safety improves Math Accuracy [7.757751847128239]
  We show that math questions, particularly those framed as natural language narratives, can serve as a subtle medium for propagating biased, unethical, or psychologically harmful content.<n>We propose SafeMath -- a safety alignment technique that reduces harmful outputs while maintaining, and in some cases improving, mathematical reasoning performance.
  arXiv  Detail & Related papers  (2026-03-26T09:06:46Z)
• PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harm [39.043933213898136]
  Current AI safety frameworks, which often treat harmfulness as binary, lack the flexibility to handle borderline cases where humans disagree.<n>We introduce PluriHarms, a benchmark designed to study human harm judgments across two key dimensions -- the harm axis (benign to harmful) and the agreement axis (agreement to disagreement)<n>Our scalable framework generates prompts that capture diverse AI harms and human values while targeting cases with high disagreement rates, validated by human data.
  arXiv  Detail & Related papers  (2026-01-13T19:41:11Z)
• ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs [48.50397204177239]
  As large language models (LLMs) evolve, evaluating the safety of their actions becomes critical.<n>We introduce ManagerBench, a benchmark that evaluates LLM decision-making in realistic, human-validated managerial scenarios.<n>A parallel control set, where potential harm is directed only at inanimate objects, measures a model's pragmatism and identifies its tendency to be overly safe.
  arXiv  Detail & Related papers  (2025-10-01T13:08:33Z)
• AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment framework [4.84912384919978]
  Existing AI risk assessment models focus on internal compliance, often neglecting diverse stakeholder perspectives and real-world consequences.<n>We propose a paradigm shift to a human-centric, harm-severity adaptive approach grounded in empirical incident data.<n>We present AI Harmonics, which includes a novel AI harm assessment metric (AIH) that leverages ordinal severity data to capture relative impact without requiring precise numerical estimates.
  arXiv  Detail & Related papers  (2025-09-12T09:52:45Z)
• Measuring Goal-Directedness [13.871986295154782]
  We define maximum entropy goal-directedness (MEG), a formal measure of goal-directedness in causal models and Markov decision processes. MEG is based on an adaptation of the maximum causal entropy framework used in inverse reinforcement learning.
  arXiv  Detail & Related papers  (2024-12-06T03:48:47Z)
• SafetyAnalyst: Interpretable, transparent, and steerable safety moderation for AI behavior [56.10557932893919]
  We present SafetyAnalyst, a novel AI safety moderation framework. Given an AI behavior, SafetyAnalyst uses chain-of-thought reasoning to analyze its potential consequences. It aggregates all harmful and beneficial effects into a harmfulness score using fully interpretable weight parameters.
  arXiv  Detail & Related papers  (2024-10-22T03:38:37Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users. We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions. We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Speculations on Uncertainty and Humane Algorithms [0.0]
  Provenance enables algorithms to know what they know preventing possible harms. It is essential to compute with what we know rather than make assumptions that may be unjustified or untenable.
  arXiv  Detail & Related papers  (2024-08-13T08:54:34Z)
• Can a Bayesian Oracle Prevent Harm from an Agent? [48.12936383352277]
  We consider estimating a context-dependent bound on the probability of violating a given safety specification. Noting that different plausible hypotheses about the world could produce very different outcomes, we derive on the safety violation probability predicted under the true but unknown hypothesis. We consider two forms of this result, in the iid case and in the non-iid case, and conclude with open problems towards turning such results into practical AI guardrails.
  arXiv  Detail & Related papers  (2024-08-09T18:10:42Z)
• Inception: Efficiently Computable Misinformation Attacks on Markov Games [14.491458698581038]
  We study security threats to Markov games due to information asymmetry and misinformation. We derive the victim's policy under worst-case rationality and present-time algorithms to compute the attacker's optimal worst-case policy. Our work exposes a security vulnerability from standard game assumptions under misinformation.
  arXiv  Detail & Related papers  (2024-06-24T20:01:43Z)
• Control Risk for Potential Misuse of Artificial Intelligence in Science [85.91232985405554]
  We aim to raise awareness of the dangers of AI misuse in science. We highlight real-world examples of misuse in chemical science. We propose a system called SciGuard to control misuse risks for AI models in science.
  arXiv  Detail & Related papers  (2023-12-11T18:50:57Z)
• Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted? [0.0]
  We argue that targeted interventions on certain capabilities will be warranted to prevent some misuses of AI. These restrictions may include controlling who can access certain types of AI models, what they can be used for, whether outputs are filtered or can be traced back to their user. We apply this reasoning to three examples: predicting novel toxins, creating harmful images, and automating spear phishing campaigns.
  arXiv  Detail & Related papers  (2023-03-16T15:05:59Z)
• A Quantitative Account of Harm [18.7822411439221]
  We first present a quantitative definition of harm in a deterministic context involving a single individual. We then consider the issues involved in dealing with uncertainty regarding the context. We show that the "obvious" way of doing this can lead to counterintuitive or inappropriate answers.
  arXiv  Detail & Related papers  (2022-09-29T21:48:38Z)
• The Hammer and the Nut: Is Bilevel Optimization Really Needed to Poison Linear Classifiers? [27.701693158702753]
  Data poisoning is a particularly worrisome subset of poisoning attacks. We propose a counter-intuitive but efficient framework to combat data poisoning. Our framework achieves comparable, or even better, performances in terms of the attacker's objective.
  arXiv  Detail & Related papers  (2021-03-23T09:08:10Z)
• Offline Contextual Bandits with Overparameterized Models [52.788628474552276]
  We ask whether the same phenomenon occurs for offline contextual bandits. We show that this discrepancy is due to the emphaction-stability of their objectives. In experiments with large neural networks, this gap between action-stable value-based objectives and unstable policy-based objectives leads to significant performance differences.
  arXiv  Detail & Related papers  (2020-06-27T13:52:07Z)
• A Deep Q-learning/genetic Algorithms Based Novel Methodology For Optimizing Covid-19 Pandemic Government Actions [63.669642197519934]
  We use the SEIR epidemiological model to represent the evolution of the virus COVID-19 over time in the population. The sequences of actions (confinement, self-isolation, two-meter distance or not taking restrictions) are evaluated according to a reward system. We prove that our methodology is a valid tool to discover actions governments can take to reduce the negative effects of a pandemic in both senses.
  arXiv  Detail & Related papers  (2020-05-15T17:17:45Z)
• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores [85.12096045419686]
  We study the adoption of an algorithmic tool used to assist child maltreatment hotline screening decisions. We first show that humans do alter their behavior when the tool is deployed. We show that humans are less likely to adhere to the machine's recommendation when the score displayed is an incorrect estimate of risk.
  arXiv  Detail & Related papers  (2020-02-19T07:27:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.