Multi-intention Inverse Q-learning for Interpretable Behavior Representation

• URL: http://arxiv.org/abs/2311.13870v4
• Date: Tue, 10 Sep 2024 10:12:56 GMT
• Title: Multi-intention Inverse Q-learning for Interpretable Behavior Representation
• Authors: Hao Zhu, Brice De La Crompe, Gabriel Kalweit, Artur Schneider, Maria Kalweit, Ilka Diester, Joschka Boedecker,
• Abstract summary: inverse reinforcement learning (IRL) methods have proven instrumental in reconstructing animal's intentions underlying complex behaviors. We introduce the class of hierarchical inverse Q-learning (HIQL) algorithms. Applying HIQL to simulated experiments and several real animal behavior datasets, our approach outperforms current benchmarks in behavior prediction.
• Score: 12.135423420992334
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In advancing the understanding of natural decision-making processes, inverse reinforcement learning (IRL) methods have proven instrumental in reconstructing animal's intentions underlying complex behaviors. Given the recent development of a continuous-time multi-intention IRL framework, there has been persistent inquiry into inferring discrete time-varying rewards with IRL. To address this challenge, we introduce the class of hierarchical inverse Q-learning (HIQL) algorithms. Through an unsupervised learning process, HIQL divides expert trajectories into multiple intention segments, and solves the IRL problem independently for each. Applying HIQL to simulated experiments and several real animal behavior datasets, our approach outperforms current benchmarks in behavior prediction and produces interpretable reward functions. Our results suggest that the intention transition dynamics underlying complex decision-making behavior is better modeled by a step function instead of a smoothly varying function. This advancement holds promise for neuroscience and cognitive science, contributing to a deeper understanding of decision-making and uncovering underlying brain mechanisms.

Related papers

• Life, uh, Finds a Way: Systematic Neural Search [2.163881720692685]
  We tackle the challenge of rapidly adapting an agent's behavior to solve continuous problems in settings. Instead of focusing on deep reinforcement learning, we propose viewing behavior as the physical manifestation of a search procedure. We describe an algorithm that implicitly enumerates behaviors by regulating the tight feedback loop between execution of behaviors and mutation of the graph.
  arXiv  Detail & Related papers  (2024-10-02T09:06:54Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Learning to Walk Autonomously via Reset-Free Quality-Diversity [73.08073762433376]
  Quality-Diversity algorithms can discover large and complex behavioural repertoires consisting of both diverse and high-performing skills. Existing QD algorithms need large numbers of evaluations as well as episodic resets, which require manual human supervision and interventions. This paper proposes Reset-Free Quality-Diversity optimization (RF-QD) as a step towards autonomous learning for robotics in open-ended environments.
  arXiv  Detail & Related papers  (2022-04-07T14:07:51Z)
• Backprop-Free Reinforcement Learning with Active Neural Generative Coding [84.11376568625353]
  We propose a computational framework for learning action-driven generative models without backpropagation of errors (backprop) in dynamic environments. We develop an intelligent agent that operates even with sparse rewards, drawing inspiration from the cognitive theory of planning as inference. The robust performance of our agent offers promising evidence that a backprop-free approach for neural inference and learning can drive goal-directed behavior.
  arXiv  Detail & Related papers  (2021-07-10T19:02:27Z)
• IQ-Learn: Inverse soft-Q Learning for Imitation [95.06031307730245]
  imitation learning from a small amount of expert data can be challenging in high-dimensional environments with complex dynamics. Behavioral cloning is a simple method that is widely used due to its simplicity of implementation and stable convergence. We introduce a method for dynamics-aware IL which avoids adversarial training by learning a single Q-function.
  arXiv  Detail & Related papers  (2021-06-23T03:43:10Z)
• Tracking Emotions: Intrinsic Motivation Grounded on Multi-Level Prediction Error Dynamics [68.8204255655161]
  We discuss how emotions arise when differences between expected and actual rates of progress towards a goal are experienced. We present an intrinsic motivation architecture that generates behaviors towards self-generated and dynamic goals.
  arXiv  Detail & Related papers  (2020-07-29T06:53:13Z)
• Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration [21.870750931559915]
  We propose a new algorithm named Wasserstein Distance guided Adrial Imitation Learning (WDAIL) for promoting the performance of imitation learning (IL) The experiment results show that the learning procedure remains remarkably stable, and achieves significant performance in the complex continuous control tasks of MuJoCo.
  arXiv  Detail & Related papers  (2020-06-05T15:10:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.