Dark Speculation: Combining Qualitative and Quantitative Understanding in Frontier AI Risk Analysis

• URL: http://arxiv.org/abs/2511.21838v1
• Date: Wed, 26 Nov 2025 19:09:16 GMT
• Title: Dark Speculation: Combining Qualitative and Quantitative Understanding in Frontier AI Risk Analysis
• Abstract summary: Estimating catastrophic harms from frontier AI is hindered by deep ambiguity.<n>We propose a process of $textitdark speculation$.<n>We formalize this process using a simplified catastrophic Lévy framework.
• Score: 39.69658116435543
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating catastrophic harms from frontier AI is hindered by deep ambiguity: many of its risks are not only unobserved but unanticipated by analysts. The central limitation of current risk analysis is the inability to populate the $\textit{catastrophic event space}$, or the set of potential large-scale harms to which probabilities might be assigned. This intractability is worsened by the $\textit{Lucretius problem}$, or the tendency to infer future risks only from past experience. We propose a process of $\textit{dark speculation}$, in which systematically generating and refining catastrophic scenarios ("qualitative" work) is coupled with estimating their likelihoods and associated damages (quantitative underwriting analysis). The idea is neither to predict the future nor to enable insurance for its own sake, but to use narrative and underwriting tools together to generate probability distributions over outcomes. We formalize this process using a simplified catastrophic Lévy stochastic framework and propose an iterative institutional design in which (1) speculation (including scenario planning) generates detailed catastrophic event narratives, (2) insurance underwriters assign probabilistic and financial parameters to these narratives, and (3) decision-makers synthesize the results into summary statistics to inform judgment. Analysis of the model reveals the value of (a) maintaining independence between speculation and underwriting, (b) analyzing multiple risk categories in parallel, and (c) generating "thick" catastrophic narrative rich in causal (counterfactual) and mitigative detail. While the approach cannot eliminate deep ambiguity, it offers a systematic approach to reason about extreme, low-probability events in frontier AI, tempering complacency and overreaction. The framework is adaptable for iterative use and can further augmented with AI systems.

Related papers

• Toward a Threat Actor Profiling Taxonomy for Pre-Release Risk Management of Open-Weight Frontier Models [2.9794937485335047]
  This capstone argues that explicit adversary characterization should be regarded as a prerequisite for evaluations that are interpretable, comparable, and faithful to the risks they target.<n>We propose a six-attribute taxonomy with empirically grounded tiers derived from existing terrorism, biosecurity, and cybersecurity literature.<n>The taxonomy is designed to function as research infrastructure: a common language for pre-specifying adversary assumptions before evaluations are conducted, analogous to pre-analysis plans for randomized controlled trials (RCTs) in medicine and economics.
  arXiv  Detail & Related papers  (2026-08-26T04:34:11Z)
• Hagenberg Risk Management Process (Part 3): Operationalization, Probabilities, and Causal Analysis [0.0]
  We bridge the gap between qualitative risk analysis and realtime risk management by extending Bowtie diagrams into a formal probabilistic runtime model.<n>Realtime Risk Studio transforms Bowtie structures into a directed acyclic graph (DAG) suitable for Bayesian inference.<n>Probability Capture automatically generates questionnaires from a DAG model so experts can provide estimates.
  arXiv  Detail & Related papers  (2026-04-10T09:40:07Z)
• Uncertainty-Aware Vision-based Risk Object Identification via Conformal Risk Tube Prediction [7.840136094141999]
  Existing approaches make decisions and ignore uncertainty, which could lead to safety-critical failures.<n>Despite these challenges, current methods lack principled framework to model risk uncertainty jointly across space and time.<n>We propose Conformal Risk Prediction unified formulation that captures principled risk uncertainty, provides coverage guarantees for true risks, and produces calibrated risk scores with uncertainty estimates.
  arXiv  Detail & Related papers  (2026-03-25T04:13:42Z)
• Conformal Thinking: Risk Control for Reasoning on a Compute Budget [60.65072883773352]
  Reasoning Large Language Models (LLMs) enable test-time scaling, with dataset-level accuracy improving as the token budget increases.<n>We re-frame the budget setting problem as risk control, limiting the error rate while minimizing compute.<n>Our framework introduces an upper threshold that stops reasoning when the model is confident and a novel lower threshold that preemptively stops unsolvable instances.
  arXiv  Detail & Related papers  (2026-02-03T18:17:22Z)
• The Role of Risk Modeling in Advanced AI Risk Management [33.357295564462284]
  Rapidly advancing artificial intelligence (AI) systems introduce novel, uncertain, and potentially catastrophic risks.<n>Managing these risks requires a mature risk-management infrastructure whose cornerstone is rigorous risk modeling.<n>We argue that advanced-AI governance should adopt a similar dual approach and that verifiable, provably-safe AI architectures are urgently needed.
  arXiv  Detail & Related papers  (2025-12-09T15:37:33Z)
• What are the odds? Risk and uncertainty about AI existential risk [0.0]
  It is not just a commentary though, but a useful reminder of the philosophical limitations of saylinear models of risk.<n>I then distinguish between risk and uncertainty, and argue that any estimation of P(D) is structurally affected by two kinds of uncertainty.
  arXiv  Detail & Related papers  (2025-10-27T15:53:23Z)
• Adapting Probabilistic Risk Assessment for AI [0.0]
  General-purpose artificial intelligence (AI) systems present an urgent risk management challenge.<n>Current methods often rely on selective testing and undocumented assumptions about risk priorities.<n>This paper introduces the probabilistic risk assessment (PRA) for AI framework.
  arXiv  Detail & Related papers  (2025-04-25T17:59:14Z)
• Risks and NLP Design: A Case Study on Procedural Document QA [52.557503571760215]
  We argue that clearer assessments of risks and harms to users will be possible when we specialize the analysis to more concrete applications and their plausible users. We conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.
  arXiv  Detail & Related papers  (2024-08-16T17:23:43Z)
• Sequential Manipulation Against Rank Aggregation: Theory and Algorithm [119.57122943187086]
  We leverage an online attack on the vulnerable data collection process. From the game-theoretic perspective, the confrontation scenario is formulated as a distributionally robust game. The proposed method manipulates the results of rank aggregation methods in a sequential manner.
  arXiv  Detail & Related papers  (2024-07-02T03:31:21Z)
• From Mean to Extreme: Formal Differential Privacy Bounds on the Success of Real-World Data Reconstruction Attacks [54.25638567385662]
  Differential Privacy in machine learning is often interpreted as guarantees against membership inference.<n> translating DP budgets into quantitative protection against the more damaging threat of data reconstruction remains a challenging open problem.<n>This paper bridges the critical gap by deriving the first formal privacy bounds tailored to the mechanics of demonstrated "from-scratch" attacks.
  arXiv  Detail & Related papers  (2024-02-20T09:52:30Z)
• Two Types of AI Existential Risk: Decisive and Accumulative [3.5051464966389116]
  This paper contrasts the conventional "decisive AI x-risk hypothesis" with an "accumulative AI x-risk hypothesis"<n>It argues that the accumulative view can reconcile seemingly incompatible perspectives on AI risks.
  arXiv  Detail & Related papers  (2024-01-15T17:06:02Z)
• RAP: Risk-Aware Prediction for Robust Planning [21.83865866611308]
  We introduce a new prediction objective to learn a risk-biased distribution over trajectories. This reduces the sample complexity of the risk estimation during online planning.
  arXiv  Detail & Related papers  (2022-10-04T04:19:15Z)
• Mitigating multiple descents: A model-agnostic framework for risk monotonization [84.6382406922369]
  We develop a general framework for risk monotonization based on cross-validation. We propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting.
  arXiv  Detail & Related papers  (2022-05-25T17:41:40Z)
• PAC$^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime [75.19403612525811]
  This work develops a multi-sample loss which can close the gap by spanning a trade-off between the two risks. Empirical study demonstrates improvement to the predictive distribution.
  arXiv  Detail & Related papers  (2020-10-19T16:08:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.