What are the odds? Risk and uncertainty about AI existential risk

• URL: http://arxiv.org/abs/2510.23453v1
• Date: Mon, 27 Oct 2025 15:53:23 GMT
• Title: What are the odds? Risk and uncertainty about AI existential risk
• Abstract summary: It is not just a commentary though, but a useful reminder of the philosophical limitations of saylinear models of risk.<n>I then distinguish between risk and uncertainty, and argue that any estimation of P(D) is structurally affected by two kinds of uncertainty.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This work is a commentary of the article \href{https://doi.org/10.18716/ojs/phai/2025.2801}{AI Survival Stories: a Taxonomic Analysis of AI Existential Risk} by Cappelen, Goldstein, and Hawthorne. It is not just a commentary though, but a useful reminder of the philosophical limitations of \say{linear} models of risk. The article will focus on the model employed by the authors: first, I discuss some differences between standard Swiss Cheese models and this one. I then argue that in a situation of epistemic indifference the probability of P(D) is higher than what one might first suggest, given the structural relationships between layers. I then distinguish between risk and uncertainty, and argue that any estimation of P(D) is structurally affected by two kinds of uncertainty: option uncertainty and state-space uncertainty. Incorporating these dimensions of uncertainty into our qualitative discussion on AI existential risk can provide a better understanding of the likeliness of P(D).

Related papers

• Subjective Risk Decomposition: A New View for Uncertainty Quantification [3.3198034832456345]
  We show how uncertainty measures can be derived via decomposition of a subjective risk.<n>Reverse cross-entropy provides a prominent example.<n>We then extend our view to learning theory.
  arXiv  Detail & Related papers  (2026-07-16T16:52:54Z)
• Trustworthy AI Suffers from Invariance Conflicts and Causality is The Solution [80.98492754957466]
  Core trustworthy AI objectives, such as fairness, robustness, privacy, and explainability, are hard to achieve simultaneously.<n>This paper argues that causality is necessary to understand and balance trade-offs in performance and multiple objectives of trustworthy AI.
  arXiv  Detail & Related papers  (2026-05-04T14:26:28Z)
• Why They Disagree: Decoding Differences in Opinions about AI Risk on the Lex Fridman Podcast [0.0]
  This paper analyzes contemporary debates about AI risk.<n>We find that differences in perspectives about existential risk ("X-risk") arise from differences in causal premises about design vs. emergence in complex systems.<n>Disagreements about these two forms of AI risk appear to share two properties: neither involves significant disagreements on moral values and both can be described in terms of differing views on the extent of boundedness of human rationality.
  arXiv  Detail & Related papers  (2025-12-06T08:48:30Z)
• Dark Speculation: Combining Qualitative and Quantitative Understanding in Frontier AI Risk Analysis [39.69658116435543]
  Estimating catastrophic harms from frontier AI is hindered by deep ambiguity.<n>We propose a process of $textitdark speculation$.<n>We formalize this process using a simplified catastrophic Lévy framework.
  arXiv  Detail & Related papers  (2025-11-26T19:09:16Z)
• Can a Bayesian Oracle Prevent Harm from an Agent? [48.12936383352277]
  We consider estimating a context-dependent bound on the probability of violating a given safety specification.<n>Noting that different plausible hypotheses about the world could produce very different outcomes, we derive on the safety violation probability predicted under the true but unknown hypothesis.<n>We consider two forms of this result, in the i.i.d. case and in the non-i.i.d. case, and conclude with open problems towards turning such theoretical results into practical AI guardrails.
  arXiv  Detail & Related papers  (2024-08-09T18:10:42Z)
• Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness [106.52630978891054]
  We present a taxonomy of uncertainty specific to vision-language AI systems. We also introduce a new metric confidence-weighted accuracy, that is well correlated with both accuracy and calibration error.
  arXiv  Detail & Related papers  (2024-07-02T04:23:54Z)
• Two Types of AI Existential Risk: Decisive and Accumulative [3.5051464966389116]
  This paper contrasts the conventional "decisive AI x-risk hypothesis" with an "accumulative AI x-risk hypothesis"<n>It argues that the accumulative view can reconcile seemingly incompatible perspectives on AI risks.
  arXiv  Detail & Related papers  (2024-01-15T17:06:02Z)
• Diagnosis Uncertain Models For Medical Risk Prediction [80.07192791931533]
  We consider a patient risk model which has access to vital signs, lab values, and prior history but does not have access to a patient's diagnosis. We show that such all-cause' risk models have good generalization across diagnoses but have a predictable failure mode. We propose a fix for this problem by explicitly modeling the uncertainty in risk prediction coming from uncertainty in patient diagnoses.
  arXiv  Detail & Related papers  (2023-06-29T23:36:04Z)
• Decision-Making Under Uncertainty: Beyond Probabilities [5.358161704743754]
  A classical assumption is that probabilities can sufficiently capture all uncertainty in a system. In this paper, the focus is on the uncertainty that goes beyond this classical interpretation. We show several solution techniques for both discrete and continuous models.
  arXiv  Detail & Related papers  (2023-03-10T10:53:33Z)
• Differentially Private Counterfactuals via Functional Mechanism [47.606474009932825]
  We propose a novel framework to generate differentially private counterfactual (DPC) without touching the deployed model or explanation set. In particular, we train an autoencoder with the functional mechanism to construct noisy class prototypes, and then derive the DPC from the latent prototypes.
  arXiv  Detail & Related papers  (2022-08-04T20:31:22Z)
• The Unreasonable Effectiveness of Deep Evidential Regression [72.30888739450343]
  A new approach with uncertainty-aware regression-based neural networks (NNs) shows promise over traditional deterministic methods and typical Bayesian NNs. We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a quantification rather than an exact uncertainty.
  arXiv  Detail & Related papers  (2022-05-20T10:10:32Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• An Imprecise Probability Approach for Abstract Argumentation based on Credal Sets [1.3764085113103217]
  We tackle the problem of calculating the degree of uncertainty of the extensions considering that the probability values of the arguments are imprecise. We use credal sets to model the uncertainty values of arguments and from these credal sets, we calculate the lower and upper bounds of the extensions.
  arXiv  Detail & Related papers  (2020-09-16T00:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.