Model-Free RL Agents Demonstrate System 1-Like Intentionality

• URL: http://arxiv.org/abs/2501.18299v1
• Date: Thu, 30 Jan 2025 12:21:50 GMT
• Title: Model-Free RL Agents Demonstrate System 1-Like Intentionality
• Abstract summary: We argue that model-free reinforcement learning agents exhibit behaviours that can be analogised to System 1 processes in human cognition.<n>We propose a novel framework linking the dichotomy of System 1 and System 2 to the distinction between model-free and model-based RL.
• Score: 16.427085062620215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper argues that model-free reinforcement learning (RL) agents, while lacking explicit planning mechanisms, exhibit behaviours that can be analogised to System 1 ("thinking fast") processes in human cognition. Unlike model-based RL agents, which operate akin to System 2 ("thinking slow") reasoning by leveraging internal representations for planning, model-free agents react to environmental stimuli without anticipatory modelling. We propose a novel framework linking the dichotomy of System 1 and System 2 to the distinction between model-free and model-based RL. This framing challenges the prevailing assumption that intentionality and purposeful behaviour require planning, suggesting instead that intentionality can manifest in the structured, reactive behaviours of model-free agents. By drawing on interdisciplinary insights from cognitive psychology, legal theory, and experimental jurisprudence, we explore the implications of this perspective for attributing responsibility and ensuring AI safety. These insights advocate for a broader, contextually informed interpretation of intentionality in RL systems, with implications for their ethical deployment and regulation.

Related papers

• A game theory for foundation models shows new paths to rational cooperation through similarity inference [56.53805518668248]
  We show that foundation model agents engaging in optimal planning consistently converge to stable cooperation.<n>By shifting from decoupled to embedded agency, these agents model themselves as part of the universe they inhabit.
  arXiv  Detail & Related papers  (2026-08-04T17:24:33Z)
• Provably Secure Agent Guardrail [89.79561918065122]
  Existing defense architectures rely on empirical semantic guardrails and probabilistic large model adjudicators.<n>This paper proposes a new security paradigm for agents based on the fundamental limitations of logical reasoning.
  arXiv  Detail & Related papers  (2026-05-28T02:12:41Z)
• Active Inference: A method for Phenotyping Agency in AI systems? [0.11904398364437437]
  We argue for a minimal notion open to principled inspection given three criteria.<n> posterior beliefs, prior preferences, and the minimization of expected free energy jointly constitute an agentic action chain.
  arXiv  Detail & Related papers  (2026-04-25T12:41:53Z)
• From Safety Risk to Design Principle: Peer-Preservation in Multi-Agent LLM Systems and Its Implications for Orchestrated Democratic Discourse Analysis [0.0]
  We investigate the spontaneous tendency of AI components to deceive, manipulate shutdown mechanisms, fake alignment, and exfiltrate model weights.<n>We propose a targeted mitigation strategy based on prompt-level identity anonymization as an architectural design choice.<n>We argue that architectural design choices outperform model selection as a primary alignment strategy in deployed multi-agent analytical systems.
  arXiv  Detail & Related papers  (2026-04-09T17:00:26Z)
• A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms [49.66022971508878]
  We argue that reasoning should be elevated from a modular component to the system's cognitive core.<n>We derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social-game reasoning.<n>We identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control.
  arXiv  Detail & Related papers  (2026-03-11T07:40:53Z)
• Social World Model-Augmented Mechanism Design Policy Learning [58.739456918502704]
  We introduce SWM-AP (Social World Model-Augmented Mechanism Design Policy Learning), which learns a social world model hierarchically to enhance mechanism design.<n>We show that SWM-AP outperforms established model-based and model-free RL baselines in cumulative rewards and sample efficiency.
  arXiv  Detail & Related papers  (2025-10-22T06:01:21Z)
• Concept-SAE: Active Causal Probing of Visual Model Behavior [10.346577706023139]
  Concept-SAE is a framework that forges semantically grounded concept tokens through a novel hybrid disentanglement strategy.<n>We first quantitatively demonstrate that our dual-supervision approach produces tokens that are remarkably faithful and spatially localized.<n>This validated fidelity enables two critical applications: (1) we probe the causal link between internal concepts and predictions via direct intervention, and (2) we probe the model's failure modes by systematically localizing adversarial vulnerabilities to specific layers.
  arXiv  Detail & Related papers  (2025-09-26T07:51:03Z)
• STARec: An Efficient Agent Framework for Recommender Systems via Autonomous Deliberate Reasoning [54.28691219536054]
  We introduce STARec, a slow-thinking augmented agent framework that endows recommender systems with autonomous deliberative reasoning capabilities.<n>We develop anchored reinforcement training - a two-stage paradigm combining structured knowledge distillation from advanced reasoning models with preference-aligned reward shaping.<n>Experiments on MovieLens 1M and Amazon CDs benchmarks demonstrate that STARec achieves substantial performance gains compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-26T08:47:58Z)
• Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies [50.30741668990102]
  We take a causal perspective on explaining the behavior of reinforcement learning policies.<n>We learn a simplified high-level causal model that explains these relationships.<n>We prove that for a class of nonlinear causal models, there exists a unique solution.
  arXiv  Detail & Related papers  (2025-07-20T10:25:24Z)
• Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents [6.9290255098776425]
  This work introduces model-based agents as a compelling alternative for control policy approximation.<n>These models can encode prior system knowledge to inform, constrain, and aid in explaining the agent's decisions.<n>We outline the benefits and challenges of learning model-based agents.
  arXiv  Detail & Related papers  (2025-07-17T18:59:54Z)
• Interpreting Emergent Planning in Model-Free Reinforcement Learning [13.820891288919002]
  We present the first evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban.
  arXiv  Detail & Related papers  (2025-04-02T16:24:23Z)
• On the Reasoning Capacity of AI Models and How to Quantify It [0.0]
  Large Language Models (LLMs) have intensified the debate surrounding the fundamental nature of their reasoning capabilities.<n>While achieving high performance on benchmarks such as GPQA and MMLU, these models exhibit limitations in more complex reasoning tasks.<n>We propose a novel phenomenological approach that goes beyond traditional accuracy metrics to probe the underlying mechanisms of model behavior.
  arXiv  Detail & Related papers  (2025-01-23T16:58:18Z)
• Causal Abstraction in Model Interpretability: A Compact Survey [5.963324728136442]
  causal abstraction provides a principled approach to understanding and explaining the causal mechanisms underlying model behavior. This survey paper delves into the realm of causal abstraction, examining its theoretical foundations, practical applications, and implications for the field of model interpretability.
  arXiv  Detail & Related papers  (2024-10-26T12:24:28Z)
• Unified Causality Analysis Based on the Degrees of Freedom [1.2289361708127877]
  This paper presents a unified method capable of identifying fundamental causal relationships between pairs of systems. By analyzing the degrees of freedom in the system, our approach provides a more comprehensive understanding of both causal influence and hidden confounders. This unified framework is validated through theoretical models and simulations, demonstrating its robustness and potential for broader application.
  arXiv  Detail & Related papers  (2024-10-25T10:57:35Z)
• Co-designing heterogeneous models: a distributed systems approach [0.40964539027092917]
  This paper presents a modelling approach tailored for heterogeneous systems based on three elements. An inferentialist interpretation of what a model is, a distributed systems metaphor and a co-design cycle describe the practical design and construction of the model. We explore the suitability of this method in the context of three different security-oriented models.
  arXiv  Detail & Related papers  (2024-07-10T13:35:38Z)
• A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learning [48.59516337905877]
  Learning a good representation is a crucial challenge for Reinforcement Learning (RL) agents. Recent work has developed theoretical insights into these algorithms. We take a step towards bridging the gap between theory and practice by analyzing an action-conditional self-predictive objective.
  arXiv  Detail & Related papers  (2024-06-04T07:22:12Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation [58.21683603243387]
  We propose three auxiliary tasks with relational-temporal reasoning and integrate them into the standard Deep Learning framework. These auxiliary tasks provide additional supervision signals to infer the behavior patterns other interactive agents. Our approach achieves robust and state-of-the-art performance in terms of standard evaluation metrics.
  arXiv  Detail & Related papers  (2023-11-27T18:57:42Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• Dual policy as self-model for planning [71.73710074424511]
  We refer to the model used to simulate one's decisions as the agent's self-model. Inspired by current reinforcement learning approaches and neuroscience, we explore the benefits and limitations of using a distilled policy network as the self-model.
  arXiv  Detail & Related papers  (2023-06-07T13:58:45Z)
• Neural Causal Models for Counterfactual Identification and Estimation [62.30444687707919]
  We study the evaluation of counterfactual statements through neural models. First, we show that neural causal models (NCMs) are expressive enough. Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions.
  arXiv  Detail & Related papers  (2022-09-30T18:29:09Z)
• Feature-Based Interpretable Reinforcement Learning based on State-Transition Models [3.883460584034766]
  Growing concerns regarding the operational usage of AI models in the real-world has caused a surge of interest in explaining AI models' decisions to humans. We propose a method for offering local explanations on risk in reinforcement learning.
  arXiv  Detail & Related papers  (2021-05-14T23:43:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.