AI Survival Stories: a Taxonomic Analysis of AI Existential Risk

• URL: http://arxiv.org/abs/2601.09765v1
• Date: Wed, 14 Jan 2026 09:13:05 GMT
• Title: AI Survival Stories: a Taxonomic Analysis of AI Existential Risk
• Authors: Herman Cappelen, Simon Goldstein, John Hawthorne,
• Abstract summary: We analyze a two premise argument that AI systems pose a threat to humanity.<n>We use these two premises to construct a taxonomy of survival stories.<n>We use our taxonomy to produce rough estimates of P(doom)
• Score: 1.2234742322758418
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Since the release of ChatGPT, there has been a lot of debate about whether AI systems pose an existential risk to humanity. This paper develops a general framework for thinking about the existential risk of AI systems. We analyze a two premise argument that AI systems pose a threat to humanity. Premise one: AI systems will become extremely powerful. Premise two: if AI systems become extremely powerful, they will destroy humanity. We use these two premises to construct a taxonomy of survival stories, in which humanity survives into the far future. In each survival story, one of the two premises fails. Either scientific barriers prevent AI systems from becoming extremely powerful; or humanity bans research into AI systems, thereby preventing them from becoming extremely powerful; or extremely powerful AI systems do not destroy humanity, because their goals prevent them from doing so; or extremely powerful AI systems do not destroy humanity, because we can reliably detect and disable systems that have the goal of doing so. We argue that different survival stories face different challenges. We also argue that different survival stories motivate different responses to the threats from AI. Finally, we use our taxonomy to produce rough estimates of P(doom), the probability that humanity will be destroyed by AI.

Related papers

• The human biological advantage over AI [0.0]
  Recent advances in AI raise the possibility that AI systems will one day do anything humans can do, only better.<n>But a deeper consideration suggests the overlooked differentiator between human beings and AI is not the brain, but the central nervous system.<n>A CNS cannot be manufactured or simulated; it must be grown as a biological construct.
  arXiv  Detail & Related papers  (2025-09-04T11:54:27Z)
• When Autonomy Breaks: The Hidden Existential Risk of AI [0.0]
  I argue that there is an underappreciated risk in the slow and irrevocable decline of human autonomy.<n>What may follow is a process of gradual de-skilling, where we lose skills that we currently take for granted.<n>The biggest threat to humanity is not that machines will become more like humans, but that humans will become more like machines.
  arXiv  Detail & Related papers  (2025-03-28T05:10:32Z)
• Evaluating Intelligence via Trial and Error [59.80426744891971]
  We introduce Survival Game as a framework to evaluate intelligence based on the number of failed attempts in a trial-and-error process.<n>When the expectation and variance of failure counts are both finite, it signals the ability to consistently find solutions to new challenges.<n>Our results show that while AI systems achieve the Autonomous Level in simple tasks, they are still far from it in more complex tasks.
  arXiv  Detail & Related papers  (2025-02-26T05:59:45Z)
• AI Consciousness and Public Perceptions: Four Futures [0.0]
  We investigate whether future human society will broadly believe advanced AI systems to be conscious. We identify four major risks: AI suffering, human disempowerment, geopolitical instability, and human depravity. The paper concludes with the main recommendations to avoid research aimed at intentionally creating conscious AI.
  arXiv  Detail & Related papers  (2024-08-08T22:01:57Z)
• Managing extreme AI risks amid rapid progress [171.05448842016125]
  We describe risks that include large-scale social harms, malicious uses, and irreversible loss of human control over autonomous AI systems. There is a lack of consensus about how exactly such risks arise, and how to manage them. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.
  arXiv  Detail & Related papers  (2023-10-26T17:59:06Z)
• AI for Mathematics: A Cognitive Science Perspective [86.02346372284292]
  Mathematics is one of the most powerful conceptual systems developed and used by the human species. Rapid progress in AI, particularly propelled by advances in large language models (LLMs), has sparked renewed, widespread interest in building such systems.
  arXiv  Detail & Related papers  (2023-10-19T02:00:31Z)
• Fairness in AI and Its Long-Term Implications on Society [68.8204255655161]
  We take a closer look at AI fairness and analyze how lack of AI fairness can lead to deepening of biases over time. We discuss how biased models can lead to more negative real-world outcomes for certain groups. If the issues persist, they could be reinforced by interactions with other risks and have severe implications on society in the form of social unrest.
  arXiv  Detail & Related papers  (2023-04-16T11:22:59Z)
• Natural Selection Favors AIs over Humans [18.750116414606698]
  We argue that the most successful AI agents will likely have undesirable traits. If such agents have intelligence that exceeds that of humans, this could lead to humanity losing control of its future. To counteract these risks and evolutionary forces, we consider interventions such as carefully designing AI agents' intrinsic motivations.
  arXiv  Detail & Related papers  (2023-03-28T17:59:12Z)
• Cybertrust: From Explainable to Actionable and Interpretable AI (AI2) [58.981120701284816]
  Actionable and Interpretable AI (AI2) will incorporate explicit quantifications and visualizations of user confidence in AI recommendations. It will allow examining and testing of AI system predictions to establish a basis for trust in the systems' decision making.
  arXiv  Detail & Related papers  (2022-01-26T18:53:09Z)
• Making AI 'Smart': Bridging AI and Cognitive Science [0.0]
  With the integration of cognitive science, the 'artificial' characteristic of Artificial Intelligence might soon be replaced with'smart' This will help develop more powerful AI systems and simultaneously gives us a better understanding of how the human brain works. We argue that the possibility of AI taking over human civilization is low as developing such an advanced system requires a better understanding of the human brain first.
  arXiv  Detail & Related papers  (2021-12-31T09:30:44Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)
• The Threat of Offensive AI to Organizations [52.011307264694665]
  This survey explores the threat of offensive AI on organizations. First, we discuss how AI changes the adversary's methods, strategies, goals, and overall attack model. Then, through a literature review, we identify 33 offensive AI capabilities which adversaries can use to enhance their attacks.
  arXiv  Detail & Related papers  (2021-06-30T01:03:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.