Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors

• URL: http://arxiv.org/abs/2507.19725v1
• Date: Sat, 26 Jul 2025 00:49:25 GMT
• Title: Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors
• Abstract summary: Intrinsic Motivation(IM) methods are an effective solution to reward-sparsity.<n>IM also causes an issue known as reward hacking' where the agent optimize for the new reward at the expense of properly playing the game.<n>This study takes a first step by empirically evaluating the impact on behavior of three IM techniques on the MiniGrid game-like environment.
• Score: 2.2169849640518153
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Games are challenging for Reinforcement Learning~(RL) agents due to their reward-sparsity, as rewards are only obtainable after long sequences of deliberate actions. Intrinsic Motivation~(IM) methods -- which introduce exploration rewards -- are an effective solution to reward-sparsity. However, IM also causes an issue known as `reward hacking' where the agent optimizes for the new reward at the expense of properly playing the game. The larger problem is that reward hacking itself is largely unknown; there is no answer to whether, and to what extent, IM rewards change the behavior of RL agents. This study takes a first step by empirically evaluating the impact on behavior of three IM techniques on the MiniGrid game-like environment. We compare these IM models with Generalized Reward Matching~(GRM), a method that can be used with any intrinsic reward function to guarantee optimality. Our results suggest that IM causes noticeable change by increasing the initial rewards, but also altering the way the agent plays; and that GRM mitigated reward hacking in some scenarios.

Related papers

• Learning Process Rewards via Success Visitation Matching for Efficient RL [54.67547527674048]
  We propose a simple approach to transform a sparse outcome reward into a dense process reward.<n>Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes.<n>By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion.
  arXiv  Detail & Related papers  (2026-06-22T17:30:24Z)
• Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking [69.06218054848803]
  We propose Adrial Reward Auditing (ARA), a framework that reconceptualizes reward hacking as a dynamic, competitive game.<n>ARA operates in two stages: first, a Hacker policy discovers reward model vulnerabilities while an Auditor learns to detect exploitation from latent representations.<n>ARA achieves the best alignment-utility tradeoff among all baselines.
  arXiv  Detail & Related papers  (2026-02-02T07:34:57Z)
• Reward Hacking Mitigation using Verifiable Composite Rewards [5.061948558533868]
  Reinforcement Learning from Verifiable Rewards (RLVR) has recently shown that large language models (LLMs) can develop their own reasoning without direct supervision.<n>This work addresses two primary forms of this behavior: i.<n>providing a final answer without preceding reasoning, and ii. employing non-standard reasoning formats to exploit the reward mechanism.
  arXiv  Detail & Related papers  (2025-09-19T03:40:27Z)
• Robust and Performance Incentivizing Algorithms for Multi-Armed Bandits with Strategic Agents [52.75161794035767]
  We introduce a class of bandit algorithms that meet the two objectives of performance incentivization and robustness simultaneously.<n>We show that settings where the principal has no information about the arms' performance characteristics can be handled by combining ideas from second price auctions with our algorithms.
  arXiv  Detail & Related papers  (2023-12-13T06:54:49Z)
• Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark [61.43264961005614]
  We develop a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios. We evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. Our results show that agents can both act competently and morally, so concrete progress can be made in machine ethics.
  arXiv  Detail & Related papers  (2023-04-06T17:59:03Z)
• Reward Design with Language Models [27.24197025688919]
  Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function.
  arXiv  Detail & Related papers  (2023-02-27T22:09:35Z)
• Redeeming Intrinsic Rewards via Constrained Optimization [17.203887958936168]
  State-of-the-art reinforcement learning (RL) algorithms typically use random sampling (e.g., $epsilon$-greedy) for exploration, but this method fails in hard exploration tasks like Montezuma's Revenge. Prior works incentivize the agent to visit novel states using an exploration bonus (also called an intrinsic reward or curiosity) Such methods can lead to excellent results on hard exploration tasks but can suffer from intrinsic reward bias and underperform when compared to an agent trained using only task rewards. We propose a principled constrained policy optimization procedure that automatically tunes the importance of the intrinsic reward.
  arXiv  Detail & Related papers  (2022-11-14T18:49:26Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards [46.068337522093096]
  We introduce the concept of motivation which captures the underlying goal of maximizing certain rewards. Our method performs better than the state-of-the-art methods in handling problems of delayed reward, exploration, and credit assignment.
  arXiv  Detail & Related papers  (2022-07-29T14:52:02Z)
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models [85.68751244243823]
  Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. We find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward.
  arXiv  Detail & Related papers  (2022-01-10T18:58:52Z)
• Explore and Control with Adversarial Surprise [78.41972292110967]
  Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-specified rewards. We propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences. We show that our method leads to the emergence of complex skills by exhibiting clear phase transitions.
  arXiv  Detail & Related papers  (2021-07-12T17:58:40Z)
• Mutual Information State Intrinsic Control [91.38627985733068]
  Intrinsically motivated RL attempts to remove this constraint by defining an intrinsic reward function. Motivated by the self-consciousness concept in psychology, we make a natural assumption that the agent knows what constitutes itself. We mathematically formalize this reward as the mutual information between the agent state and the surrounding state.
  arXiv  Detail & Related papers  (2021-03-15T03:03:36Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)
• Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games [0.0]
  Training agents using Reinforcement Learning in games with sparse rewards is a challenging problem. We present a novel technique that successfully trains agents to eventually optimize the true objective in games with sparse rewards.
  arXiv  Detail & Related papers  (2020-10-05T03:43:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.