Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power

• URL: http://arxiv.org/abs/2508.00159v2
• Date: Mon, 04 Aug 2025 21:59:37 GMT
• Title: Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power
• Authors: Jobst Heitzig, Ram Potham,
• Abstract summary: This paper explores the idea of promoting both safety and wellbeing by forcing AI agents explicitly to empower humans.<n>We design a parametrizable and decomposable objective function that represents an inequality- and risk-averse long-term aggregate of human power.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Power is a key concept in AI safety: power-seeking as an instrumental goal, sudden or gradual disempowerment of humans, power balance in human-AI interaction and international AI governance. At the same time, power as the ability to pursue diverse goals is essential for wellbeing. This paper explores the idea of promoting both safety and wellbeing by forcing AI agents explicitly to empower humans and to manage the power balance between humans and AI agents in a desirable way. Using a principled, partially axiomatic approach, we design a parametrizable and decomposable objective function that represents an inequality- and risk-averse long-term aggregate of human power. It takes into account humans' bounded rationality and social norms, and, crucially, considers a wide variety of possible human goals. We derive algorithms for computing that metric by backward induction or approximating it via a form of multi-agent reinforcement learning from a given world model. We exemplify the consequences of (softly) maximizing this metric in a variety of paradigmatic situations and describe what instrumental sub-goals it will likely imply. Our cautious assessment is that softly maximizing suitable aggregate metrics of human power might constitute a beneficial objective for agentic AI systems that is safer than direct utility-based objectives.

Related papers

• Measurement of LLM's Philosophies of Human Nature [113.47929131143766]
  We design the standardized psychological scale specifically targeting large language models (LLM)<n>We show that current LLMs exhibit a systemic lack of trust in humans.<n>We propose a mental loop learning framework, which enables LLM to continuously optimize its value system.
  arXiv  Detail & Related papers  (2025-04-03T06:22:19Z)
• Universal AI maximizes Variational Empowerment [0.0]
  We build on the existing framework of Self-AIXI -- a universal learning agent that predicts its own actions.<n>We argue that power-seeking tendencies of universal AI agents can be explained as an instrumental strategy to secure future reward.<n>Our main contribution is to show how these motivations systematically lead universal AI agents to seek and sustain high-optionality states.
  arXiv  Detail & Related papers  (2025-02-20T02:58:44Z)
• A Beautiful Mind: Principles and Strategies for AI-Augmented Human Reasoning [0.0]
  This paper outlines a human-centered augmented reasoning paradigm.<n>It offers examples of interaction modes that can serve as bridges between human reasoning and AI algorithms.
  arXiv  Detail & Related papers  (2025-02-05T20:57:29Z)
• Aligning Generalisation Between Humans and Machines [74.120848518198]
  AI technology can support humans in scientific discovery and forming decisions, but may also disrupt democracies and target individuals.<n>The responsible use of AI and its participation in human-AI teams increasingly shows the need for AI alignment.<n>A crucial yet often overlooked aspect of these interactions is the different ways in which humans and machines generalise.
  arXiv  Detail & Related papers  (2024-11-23T18:36:07Z)
• Human-AI Safety: A Descendant of Generative AI and Control Systems Safety [6.100304850888953]
  We argue that meaningful safety assurances for advanced AI technologies require reasoning about how the feedback loop formed by AI outputs and human behavior may drive the interaction towards different outcomes. We propose a concrete technical roadmap towards next-generation human-centered AI safety.
  arXiv  Detail & Related papers  (2024-05-16T03:52:00Z)
• Position Paper: Agent AI Towards a Holistic Intelligence [53.35971598180146]
  We emphasize developing Agent AI -- an embodied system that integrates large foundation models into agent actions. In this paper, we propose a novel large action model to achieve embodied intelligent behavior, the Agent Foundation Model.
  arXiv  Detail & Related papers  (2024-02-28T16:09:56Z)
• Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Tasks [25.507656595628376]
  We introduce a framework in which an AI agent intervenes on the parameters of a Markov Decision Process (MDP) belonging to a boundedly rational human agent. We show that AI planning with our human models can lead to helpful policies on a wide range of more complex, ground-truth humans.
  arXiv  Detail & Related papers  (2024-01-26T14:59:48Z)
• Exploration with Principles for Diverse AI Supervision [88.61687950039662]
  Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. We propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data.
  arXiv  Detail & Related papers  (2023-10-13T07:03:39Z)
• The Rise and Potential of Large Language Model Based Agents: A Survey [91.71061158000953]
  Large language models (LLMs) are regarded as potential sparks for Artificial General Intelligence (AGI) We start by tracing the concept of agents from its philosophical origins to its development in AI, and explain why LLMs are suitable foundations for agents. We explore the extensive applications of LLM-based agents in three aspects: single-agent scenarios, multi-agent scenarios, and human-agent cooperation.
  arXiv  Detail & Related papers  (2023-09-14T17:12:03Z)
• Learning Complementary Policies for Human-AI Teams [22.13683008398939]
  We propose a framework for a novel human-AI collaboration for selecting advantageous course of action. Our solution aims to exploit the human-AI complementarity to maximize decision rewards.
  arXiv  Detail & Related papers  (2023-02-06T17:22:18Z)
• A Cognitive Framework for Delegation Between Error-Prone AI and Human Agents [0.0]
  We investigate the use of cognitively inspired models of behavior to predict the behavior of both human and AI agents. The predicted behavior is used to delegate control between humans and AI agents through the use of an intermediary entity.
  arXiv  Detail & Related papers  (2022-04-06T15:15:21Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.