AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions

• URL: http://arxiv.org/abs/2505.04592v1
• Date: Wed, 07 May 2025 17:35:36 GMT
• Title: AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions
• Authors: Peter Barnett, Aaron Scher,
• Abstract summary: Humanity appears to be on course to soon develop AI systems that substantially outperform human experts.<n>We believe the default trajectory has a high likelihood of catastrophe, including human extinction.<n>Risks come from failure to control powerful AI systems, misuse of AI by malicious rogue actors, war between great powers, and authoritarian lock-in.
• Score: 2.07180164747172
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humanity appears to be on course to soon develop AI systems that substantially outperform human experts in all cognitive domains and activities. We believe the default trajectory has a high likelihood of catastrophe, including human extinction. Risks come from failure to control powerful AI systems, misuse of AI by malicious rogue actors, war between great powers, and authoritarian lock-in. This research agenda has two aims: to describe the strategic landscape of AI development and to catalog important governance research questions. These questions, if answered, would provide important insight on how to successfully reduce catastrophic risks. We describe four high-level scenarios for the geopolitical response to advanced AI development, cataloging the research questions most relevant to each. Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch), which leads into an internationally coordinated Halt on frontier AI activities at some point in the future. The second scenario we describe is a US National Project for AI, in which the US Government races to develop advanced AI systems and establish unilateral control over global AI development. We also describe two additional scenarios: a Light-Touch world similar to that of today and a Threat of Sabotage situation where countries use sabotage and deterrence to slow AI development. In our view, apart from the Off Switch and Halt scenario, all of these trajectories appear to carry an unacceptable risk of catastrophic harm. Urgent action is needed from the US National Security community and AI governance ecosystem to answer key research questions, build the capability to halt dangerous AI activities, and prepare for international AI agreements.

Related papers

• The Singapore Consensus on Global AI Safety Research Priorities [128.58674892183657]
  "2025 Singapore Conference on AI (SCAI): International Scientific Exchange on AI Safety" aimed to support research in this space.<n>Report builds on the International AI Safety Report chaired by Yoshua Bengio and backed by 33 governments.<n>Report organises AI safety research domains into three types: challenges with creating trustworthy AI systems (Development), challenges with evaluating their risks (Assessment) and challenges with monitoring and intervening after deployment (Control)
  arXiv  Detail & Related papers  (2025-06-25T17:59:50Z)
• Exploiting AI for Attacks: On the Interplay between Adversarial AI and Offensive AI [18.178555463870214]
  AI as a target of attacks (Adversarial AI') and AI as a means to launch attacks on any target (Offensive AI')<n>This article explores two emerging AI-related threats and the interplay between them.
  arXiv  Detail & Related papers  (2025-06-14T14:21:01Z)
• Who is Responsible When AI Fails? Mapping Causes, Entities, and Consequences of AI Privacy and Ethical Incidents [29.070947259551478]
  We analyzed 202 real-world AI privacy and ethical incidents.<n>This produced a taxonomy that classifies incident types across AI lifecycle stages.<n>It accounts for contextual factors such as causes, responsible entities, disclosure sources, and impacts.
  arXiv  Detail & Related papers  (2025-03-28T21:57:38Z)
• AI threats to national security can be countered through an incident regime [55.2480439325792]
  We propose a legally mandated post-deployment AI incident regime that aims to counter potential national security threats from AI systems.<n>Our proposed AI incident regime is split into three phases. The first phase revolves around a novel operationalization of what counts as an 'AI incident'<n>The second and third phases spell out that AI providers should notify a government agency about incidents, and that the government agency should be involved in amending AI providers' security and safety procedures.
  arXiv  Detail & Related papers  (2025-03-25T17:51:50Z)
• Superintelligence Strategy: Expert Version [64.7113737051525]
  Destabilizing AI developments could raise the odds of great-power conflict.<n>Superintelligence -- AI vastly better than humans at nearly all cognitive tasks -- is now anticipated by AI researchers.<n>We introduce the concept of Mutual Assured AI Malfunction.
  arXiv  Detail & Related papers  (2025-03-07T17:53:24Z)
• Ten Hard Problems in Artificial Intelligence We Must Get Right [72.99597122935903]
  We explore the AI2050 "hard problems" that block the promise of AI and cause AI risks. For each problem, we outline the area, identify significant recent work, and suggest ways forward.
  arXiv  Detail & Related papers  (2024-02-06T23:16:41Z)
• Taking control: Policies to address extinction risks from AI [0.0]
  We argue that voluntary commitments from AI companies would be an inappropriate and insufficient response. We describe three policy proposals that would meaningfully address the threats from advanced AI.
  arXiv  Detail & Related papers  (2023-10-31T15:53:14Z)
• Managing extreme AI risks amid rapid progress [171.05448842016125]
  We describe risks that include large-scale social harms, malicious uses, and irreversible loss of human control over autonomous AI systems. There is a lack of consensus about how exactly such risks arise, and how to manage them. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.
  arXiv  Detail & Related papers  (2023-10-26T17:59:06Z)
• Proceedings of the Artificial Intelligence for Cyber Security (AICS) Workshop at AAAI 2022 [55.573187938617636]
  The workshop will focus on the application of AI to problems in cyber security. Cyber systems generate large volumes of data, utilizing this effectively is beyond human capabilities.
  arXiv  Detail & Related papers  (2022-02-28T18:27:41Z)
• Socially Responsible AI Algorithms: Issues, Purposes, and Challenges [31.382000425295885]
  Technologists and AI researchers have a responsibility to develop trustworthy AI systems. To build long-lasting trust between AI and human beings, we argue that the key is to think beyond algorithmic fairness.
  arXiv  Detail & Related papers  (2021-01-01T17:34:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.