Task-Induced Representational Invariances Depend on Learning Objective in Deep RL

• URL: http://arxiv.org/abs/2606.01868v1
• Date: Mon, 01 Jun 2026 08:16:48 GMT
• Title: Task-Induced Representational Invariances Depend on Learning Objective in Deep RL
• Abstract summary: Reinforcement Learning (RL) has long served as a model for goal-directed animal behavior in neuroscience.<n>We analyze deep RL representations through the lens of MDP reduction theory.<n>Our findings provide a principled approach to comparing learned representations across RL algorithms, with demonstrated practical implications and possible insights for neural coding in the brain.
• Score: 7.985354141014646
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) has long served as a model for goal-directed animal behavior in neuroscience. Modern deep RL has shown remarkable success across many domains, further strengthening this connection. The ability to learn abstract representations of high-dimensional state spaces underlies much of this success. However, theoretical understanding of these learned representations remains limited, hindering direct comparisons between models and animal learning. We address this gap by analyzing deep RL representations through the lens of MDP reduction theory. Investigating canonical RL algorithms in a navigation task, we find that even when performance is comparable, the value-based method (DQN) learns representations that are invariant to MDP homomorphism symmetries, while the policy-gradient method (PPO) learns representations invariant to action symmetries. These differences emerge consistently across domains, have downstream consequences for transfer learning, and appear in LLMs in a prompt-dependent manner. Our findings provide a principled approach to comparing learned representations across RL algorithms, with demonstrated practical implications and possible insights for neural coding in the brain.

Related papers

• SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs [55.96262261252617]
  Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs)<n>Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks.<n>We propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.
  arXiv  Detail & Related papers  (2026-08-04T12:32:26Z)
• Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer [0.9558392439655014]
  We propose a novel deep Q-learning approach that introduces a regularization term to reduce positive correlations between feature representation of states.<n>We demonstrate the efficacy of our approach in improving transfer learning performance and thereby reducing computational overhead.
  arXiv  Detail & Related papers  (2025-09-29T15:44:35Z)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning [93.00629872970364]
  Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks.<n>We introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions.<n>We study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training.
  arXiv  Detail & Related papers  (2025-06-05T07:53:59Z)
• The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions [13.774600272141761]
  Reinforcement learning algorithms have proven transformative in a range of domains.<n>Much theory of RL has focused on discrete state spaces or worst-case analysis.<n>We propose a solvable high-dimensional model of RL that can capture a variety of learning protocols.
  arXiv  Detail & Related papers  (2023-06-17T18:16:51Z)
• Disentangled Representation Learning [46.51815065323667]
  Disentangled Representation Learning (DRL) aims to learn a model capable of identifying and disentangling the underlying factors hidden in the observable data in representation form. We comprehensively investigate DRL from various aspects including motivations, definitions, methodologies, evaluations, applications, and model designs.
  arXiv  Detail & Related papers  (2022-11-21T18:14:38Z)
• What deep reinforcement learning tells us about human motor learning and vice-versa [24.442174952832108]
  We show how recent deep RL methods correspond to the dominant motor learning framework in neuroscience, error-based learning. We introduce a novel deep RL algorithm: model-based deterministic policy gradients (MB-DPG) MB-DPG draws inspiration from error-based learning by explicitly relying on the observed outcome of actions.
  arXiv  Detail & Related papers  (2022-08-23T11:56:49Z)
• Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning [92.18524491615548]
  Contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL) We study how RL can be empowered by contrastive learning in a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. Under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs.
  arXiv  Detail & Related papers  (2022-07-29T17:29:08Z)
• Contrastive Learning as Goal-Conditioned Reinforcement Learning [147.28638631734486]
  In reinforcement learning (RL), it is easier to solve a task if given a good representation. While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable. We show (contrastive) representation learning methods can be cast as RL algorithms in their own right.
  arXiv  Detail & Related papers  (2022-06-15T14:34:15Z)
• Exploratory State Representation Learning [63.942632088208505]
  We propose a new approach called XSRL (eXploratory State Representation Learning) to solve the problems of exploration and SRL in parallel. On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations. On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the objective of a discovery policy.
  arXiv  Detail & Related papers  (2021-09-28T10:11:07Z)
• FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs [53.710405006523274]
  This work focuses on the representation learning question: how can we learn such features? Under the assumption that the underlying (unknown) dynamics correspond to a low rank transition matrix, we show how the representation learning question is related to a particular non-linear matrix decomposition problem. We develop FLAMBE, which engages in exploration and representation learning for provably efficient RL in low rank transition models.
  arXiv  Detail & Related papers  (2020-06-18T19:11:18Z)
• Graph-based State Representation for Deep Reinforcement Learning [1.5901689240516976]
  We exploit the fact that the underlying Markov decision process (MDP) represents a graph, which enables us to incorporate the topological information for effective state representation learning. Motivated by the recent success of node representations for several graph analytical tasks we specifically investigate the capability of node representation learning methods to effectively encode the topology of the underlying MDP in Deep RL. We find that all embedding methods outperform the commonly used matrix representation of grid-world environments in all of the studied cases.
  arXiv  Detail & Related papers  (2020-04-29T05:43:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.