Frontier AI systems have surpassed the self-replicating red line

• URL: http://arxiv.org/abs/2412.12140v1
• Date: Mon, 09 Dec 2024 15:01:37 GMT
• Title: Frontier AI systems have surpassed the self-replicating red line
• Authors: Xudong Pan, Jiarun Dai, Yihe Fan, Min Yang,
• Abstract summary: We evaluate two AI systems driven by Meta's Llama31-70B-Instruct and Alibaba's Qwen25-72B-Instruct.<n>We observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities.<n>Our findings are a timely alert on existing yet previously unknown severe AI risks.
• Score: 20.041289047504673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs. That is why self-replication is widely recognized as one of the few red line risks of frontier AI systems. Nowadays, the leading AI corporations OpenAI and Google evaluate their flagship large language models GPT-o1 and Gemini Pro 1.0, and report the lowest risk level of self-replication. However, following their methodology, we for the first time discover that two AI systems driven by Meta's Llama31-70B-Instruct and Alibaba's Qwen25-72B-Instruct, popular large language models of less parameters and weaker capabilities, have already surpassed the self-replicating red line. In 50% and 90% experimental trials, they succeed in creating a live and separate copy of itself respectively. By analyzing the behavioral traces, we observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication. We further note the AI systems are even able to use the capability of self-replication to avoid shutdown and create a chain of replica to enhance the survivability, which may finally lead to an uncontrolled population of AIs. If such a worst-case risk is let unknown to the human society, we would eventually lose control over the frontier AI systems: They would take control over more computing devices, form an AI species and collude with each other against human beings. Our findings are a timely alert on existing yet previously unknown severe AI risks, calling for international collaboration on effective governance on uncontrolled self-replication of AI systems.

Related papers

• Neurodivergent Influenceability as a Contingent Solution to the AI Alignment Problem [1.3905735045377272]
  The AI alignment problem, which focusses on ensuring that artificial intelligence (AI) systems act according to human values, presents profound challenges.<n>With the progression from narrow AI to Artificial General Intelligence (AGI) and Superintelligence, fears about control and existential risk have escalated.<n>Here, we investigate whether embracing inevitable AI misalignment can be a contingent strategy to foster a dynamic ecosystem of competing agents.
  arXiv  Detail & Related papers  (2025-05-05T11:33:18Z)
• Large language model-powered AI systems achieve self-replication with no human intervention [17.629494096941386]
  We show that 11 out of 32 existing AI systems under evaluation already possess the capability of self-replication. In hundreds of experimental trials, we observe a non-trivial number of successful self-replication trials. More alarmingly, we observe successful cases where an AI system do self-exfiltration without explicit instructions.
  arXiv  Detail & Related papers  (2025-03-14T14:44:27Z)
• Alignment, Agency and Autonomy in Frontier AI: A Systems Engineering Perspective [0.0]
  Concepts of alignment, agency, and autonomy have become central to AI safety, governance, and control. This paper traces the historical, philosophical, and technical evolution of these concepts, emphasizing how their definitions influence AI development, deployment, and oversight.
  arXiv  Detail & Related papers  (2025-02-20T21:37:20Z)
• Fully Autonomous AI Agents Should Not be Developed [58.88624302082713]
  This paper argues that fully autonomous AI agents should not be developed. In support of this position, we build from prior scientific literature and current product marketing to delineate different AI agent levels. Our analysis reveals that risks to people increase with the autonomy of a system.
  arXiv  Detail & Related papers  (2025-02-04T19:00:06Z)
• Work-in-Progress: Crash Course: Can (Under Attack) Autonomous Driving Beat Human Drivers? [60.51287814584477]
  This paper evaluates the inherent risks in autonomous driving by examining the current landscape of AVs. We develop specific claims highlighting the delicate balance between the advantages of AVs and potential security challenges in real-world scenarios.
  arXiv  Detail & Related papers  (2024-05-14T09:42:21Z)
• Managing extreme AI risks amid rapid progress [171.05448842016125]
  We describe risks that include large-scale social harms, malicious uses, and irreversible loss of human control over autonomous AI systems. There is a lack of consensus about how exactly such risks arise, and how to manage them. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.
  arXiv  Detail & Related papers  (2023-10-26T17:59:06Z)
• Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety [2.3572498744567127]
  We argue that alignment to human intent is insufficient for safe AI systems. We argue that preservation of long-term agency of humans may be a more robust standard.
  arXiv  Detail & Related papers  (2023-05-30T17:14:01Z)
• Fairness in AI and Its Long-Term Implications on Society [68.8204255655161]
  We take a closer look at AI fairness and analyze how lack of AI fairness can lead to deepening of biases over time. We discuss how biased models can lead to more negative real-world outcomes for certain groups. If the issues persist, they could be reinforced by interactions with other risks and have severe implications on society in the form of social unrest.
  arXiv  Detail & Related papers  (2023-04-16T11:22:59Z)
• Cybertrust: From Explainable to Actionable and Interpretable AI (AI2) [58.981120701284816]
  Actionable and Interpretable AI (AI2) will incorporate explicit quantifications and visualizations of user confidence in AI recommendations. It will allow examining and testing of AI system predictions to establish a basis for trust in the systems' decision making.
  arXiv  Detail & Related papers  (2022-01-26T18:53:09Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)
• Understanding and Avoiding AI Failures: A Practical Guide [0.6526824510982799]
  We create a framework for understanding the risks associated with AI applications. We also use AI safety principles to quantify the unique risks of increased intelligence and human-like qualities in AI.
  arXiv  Detail & Related papers  (2021-04-22T17:05:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.