Performance of Bounded-Rational Agents With the Ability to Self-Modify

• URL: http://arxiv.org/abs/2011.06275v2
• Date: Mon, 18 Jan 2021 09:55:26 GMT
• Title: Performance of Bounded-Rational Agents With the Ability to Self-Modify
• Abstract summary: Self-modification of agents embedded in complex environments is hard to avoid. It has been argued that intelligent agents have an incentive to avoid modifying their utility function so that their future instances work towards the same goals. We show that this result is no longer true for agents with bounded rationality.
• Score: 1.933681537640272
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-modification of agents embedded in complex environments is hard to avoid, whether it happens via direct means (e.g. own code modification) or indirectly (e.g. influencing the operator, exploiting bugs or the environment). It has been argued that intelligent agents have an incentive to avoid modifying their utility function so that their future instances work towards the same goals. Everitt et al. (2016) formally show that providing an option to self-modify is harmless for perfectly rational agents. We show that this result is no longer true for agents with bounded rationality. In such agents, self-modification may cause exponential deterioration in performance and gradual misalignment of a previously aligned agent. We investigate how the size of this effect depends on the type and magnitude of imperfections in the agent's rationality (1-4 below). We also discuss model assumptions and the wider problem and framing space. We examine four ways in which an agent can be bounded-rational: it either (1) doesn't always choose the optimal action, (2) is not perfectly aligned with human values, (3) has an inaccurate model of the environment, or (4) uses the wrong temporal discounting factor. We show that while in the cases (2)-(4) the misalignment caused by the agent's imperfection does not increase over time, with (1) the misalignment may grow exponentially.

Related papers

• When Regulation Has Memory: Hysteresis and Control Burden in Artificial Agency [0.0]
  Two systems may reach similar internal states while one requires much more corrective control to get there.<n>Using a computational model of adaptive uncertainty regulation, we study whether that burden depends on history.<n>Results suggest that adaptive agents should be evaluated not only by whether they remain organized, but by how much regulation they must recruit to do so.
  arXiv  Detail & Related papers  (2026-06-29T23:19:26Z)
• HypoAgent: An Agentic Framework for Interactive Abductive Hypothesis Generation over Knowledge Graphs [52.91976391891331]
  Abductive reasoning over knowledge graphs aims to generate logical hypotheses that explain observed entities or facts.<n>Existing controllable hypothesis generation methods allow users to guide this process with explicit conditions.<n>We propose HypoAgent, an Agentic framework for interactive abductive Hypothesis Generation over knowledge graphs.
  arXiv  Detail & Related papers  (2026-05-29T14:40:37Z)
• M2A: Synergizing Mathematical and Agentic Reasoning in Large Language Models [9.712139528293946]
  We propose M2A, a novel paradigm that synergizes mathematical and agentic reasoning via model merging.<n>M2A operates directly in parameter space, identifies the feature subspace critical for agent behavior, and merges the mathematical reasoning task vector only along its null space.<n>Unlike SFT or RL, M2A requires no additional gradient-update and exposes the coefficient merging as a simple knob for controlling reasoning length.
  arXiv  Detail & Related papers  (2026-05-11T02:05:30Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Real-Time Reasoning Agents in Evolving Environments [52.21796134114843]
  We introduce real-time reasoning as a new problem formulation for agents in evolving environments.<n>Our work establishes real-time reasoning as a critical testbed for developing practical agents.
  arXiv  Detail & Related papers  (2025-11-07T00:51:02Z)
• The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy [9.553819152637493]
  We study a minimal control interface where an agent chooses whether to act autonomously (play) or defer (ask)<n>If the agent defers, the human's choice determines the outcome, potentially leading to a corrective action or a system shutdown.<n>Our analysis focuses on cases where this game qualifies as a Markov Potential Game (MPG), a class of games where we can provide an alignment guarantee.
  arXiv  Detail & Related papers  (2025-10-30T17:46:49Z)
• AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents [0.0]
  We introduce a misalignment propensity benchmark, AgentMisalignment, consisting of a suite of realistic scenarios.<n>We organise our evaluations into subcategories of misaligned behaviours, including goal-guarding, resisting shutdown, sandbagging, and power-seeking.<n>We report the performance of frontier models on our benchmark, observing higher misalignment on average when evaluating more capable models.
  arXiv  Detail & Related papers  (2025-06-04T14:46:47Z)
• The Limits of Predicting Agents from Behaviour [16.80911584745046]
  We provide a precise answer under the assumption that the agent's behaviour is guided by a world model.<n>Our contribution is the derivation of novel bounds on the agent's behaviour in new (unseen) deployment environments.<n>We discuss the implications of these results for several research areas including fairness and safety.
  arXiv  Detail & Related papers  (2025-06-03T14:24:58Z)
• Toward a Theory of Agents as Tool-Use Decision-Makers [89.26889709510242]
  We argue that true autonomy requires agents to be grounded in a coherent epistemic framework that governs what they know, what they need to know, and how to acquire that knowledge efficiently.<n>We propose a unified theory that treats internal reasoning and external actions as equivalent epistemic tools, enabling agents to systematically coordinate introspection and interaction.<n>This perspective shifts the design of agents from mere action executors to knowledge-driven intelligence systems, offering a principled path toward building foundation agents capable of adaptive, efficient, and goal-directed behavior.
  arXiv  Detail & Related papers  (2025-06-01T07:52:16Z)
• Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting [64.13583792391783]
  inverse reinforcement learning aims to infer an agent's preferences from observing their behaviour. One of the central difficulties in IRL is that multiple preferences may lead to the same observed behaviour. We show that generally IRL is unable to infer enough information about $R$ to identify the correct optimal policy.
  arXiv  Detail & Related papers  (2024-12-15T11:08:58Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• Agents Need Not Know Their Purpose [0.0]
  This paper describes oblivious agents: agents architected in such a way that their effective utility function is an aggregation of hidden sub-functions. We show that an oblivious agent, behaving rationally, constructs an internal approximation of designers' intentions.
  arXiv  Detail & Related papers  (2024-02-15T06:15:46Z)
• On the Convergence of Bounded Agents [80.67035535522777]
  A bounded agent has converged when the minimal number of states needed to describe the agent's future behavior cannot decrease. The second view says that a bounded agent has converged just when the agent's performance only changes if the agent's internal state changes.
  arXiv  Detail & Related papers  (2023-07-20T17:27:29Z)
• Decision-Making Among Bounded Rational Agents [5.24482648010213]
  We introduce the concept of bounded rationality from an information-theoretic view into the game-theoretic framework. This allows the robots to reason other agents' sub-optimal behaviors and act accordingly under their computational constraints. We demonstrate that the resulting framework allows the robots to reason about different levels of rational behaviors of other agents and compute a reasonable strategy under its computational constraint.
  arXiv  Detail & Related papers  (2022-10-17T00:29:24Z)
• On Avoiding Power-Seeking by Artificial Intelligence [93.9264437334683]
  We do not know how to align a very intelligent AI agent's behavior with human interests. I investigate whether we can build smart AI agents which have limited impact on the world, and which do not autonomously seek power.
  arXiv  Detail & Related papers  (2022-06-23T16:56:21Z)
• Formalizing the Problem of Side Effect Regularization [81.97441214404247]
  We propose a formal criterion for side effect regularization via the assistance game framework. In these games, the agent solves a partially observable Markov decision process. We show that this POMDP is solved by trading off the proxy reward with the agent's ability to achieve a range of future tasks.
  arXiv  Detail & Related papers  (2022-06-23T16:36:13Z)
• Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty [54.88405167739227]
  We present HAICU, a method for heterogeneous-agent trajectory forecasting that explicitly incorporates agents' class probabilities. We additionally present PUP, a new challenging real-world autonomous driving dataset. We demonstrate that incorporating class probabilities in trajectory forecasting significantly improves performance in the face of uncertainty.
  arXiv  Detail & Related papers  (2021-04-26T10:28:34Z)
• Empirically Verifying Hypotheses Using Reinforcement Learning [58.09414653169534]
  This paper formulates hypothesis verification as an RL problem. We aim to build an agent that, given a hypothesis about the dynamics of the world, can take actions to generate observations which can help predict whether the hypothesis is true or false.
  arXiv  Detail & Related papers  (2020-06-29T01:01:10Z)
• Pessimism About Unknown Unknowns Inspires Conservatism [24.085795452335145]
  We define an idealized Bayesian reinforcement learner which follows a policy that maximizes the worst-case expected reward over a set of world-models. A scalar parameter tunes the agent's pessimism by changing the size of the set of world-models taken into account. Since pessimism discourages exploration, at each timestep, the agent may defer to a mentor, who may be a human or some known-safe policy.
  arXiv  Detail & Related papers  (2020-06-15T20:46:33Z)
• Distributing entanglement with separable states: assessment of encoding and decoding imperfections [55.41644538483948]
  Entanglement can be distributed using a carrier which is always separable from the rest of the systems involved. We consider the effect of incoherent dynamics acting alongside imperfect unitary interactions. We show that entanglement gain is possible even with substantial unitary errors.
  arXiv  Detail & Related papers  (2020-02-11T15:25:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.