NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learning

• URL: http://arxiv.org/abs/2606.21297v1
• Date: Fri, 19 Jun 2026 10:21:04 GMT
• Title: NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learning
• Abstract summary: Augmenting model-free reinforcement learning (RL) with representations learned through observation dynamics prediction (observation-predictive RL) can improve sample efficiency and performance.<n>We propose a novel normalization method tailored to online RL, which normalizes low-dimensional observations and balances the resulting losses and gradients.<n>We introduce Normalized Observation Space Dynamics-Augmented Q-learning ( NASDAQ), a framework for observation-predictive RL applicable across diverse domains.
• Score: 13.486549278186928
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Augmenting model-free reinforcement learning (RL) with representations learned through observation dynamics prediction (observation-predictive RL) can improve sample efficiency and performance, with minor modifications and limited additional computation. However, this approach still struggles in challenging tasks with low-dimensional observations. In this paper, we identify a key factor behind this problem: unbalanced reconstruction losses across observation dimensions, where dimensions with larger value ranges dominate the loss. This encourages the agent to neglect dimensions with relatively small ranges, leading to degraded performance. To address this issue, we propose a novel normalization method tailored to online RL, which normalizes low-dimensional observations and balances the resulting losses and gradients. Beyond balancing reconstruction losses, observation normalization enables dynamics prediction to be performed in a normalized observation space, thereby providing a unified treatment of low- and high-dimensional inputs (e.g., physical states and images). Building on this idea, we further introduce Normalized Observation Space Dynamics-Augmented Q-learning (NASDAQ), a framework for observation-predictive RL applicable across diverse domains. NASDAQ learns state-action representations by coupling value learning with two auxiliary tasks: short-term value prediction and next normalized observation prediction. Extensive experiments demonstrate that NASDAQ achieves competitive or superior performance compared with state-of-the-art model-based and self-predictive RL methods, while requiring significantly less training wall-time.

Related papers

• Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control [10.054153317469096]
  We propose Observation-Grounded Self-Predictive Representations (OG-SPR) for continuous control.<n>OG-SPR incorporates two core auxiliary objectives: multi-step latent self-prediction and next-observation prediction.<n> Experiments show that OG-SPR improves aggregate performance over state-of-the-art self-predictive and observation-predictive RL methods.
  arXiv  Detail & Related papers  (2026-08-06T13:02:59Z)
• Convergent Stochastic Training of Attention and Understanding LoRA [40.81245220414135]
  Transformers have revolutionized machine learning and deploying attention layers in the model is increasingly standard across a myriad of applications.<n>For large models, it is common to implement Low Rank Adaptation (LoRA), whereby a factorized parameterization of them is trained, to achieve a surprisingly beneficial accuracy-size tradeoff.<n>We prove that for any mild regularization, the empirical regression loss on a attention layer and LoRA on a shallow neural net induce Poincaré inequality for the corresponding Gibbs' measure.
  arXiv  Detail & Related papers  (2026-05-08T16:22:08Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Revisiting Entropy in Reinforcement Learning for Large Reasoning Models [54.96908589622163]
  We investigate the entropy dynamics of large language models trained withReinforcement learning with verifiable rewards (RLVR)<n>Our findings reveal that the number of off-policy updates, the diversity of training data, and the clipping thresholds in the optimization objective are critical factors influencing the entropy of LLMs trained with RLVR.
  arXiv  Detail & Related papers  (2025-11-08T12:50:41Z)
• VIKING: Deep variational inference with stochastic projections [48.946143517489496]
  Variational mean field approximations tend to struggle with contemporary overparametrized deep neural networks.<n>We propose a simple variational family that considers two independent linear subspaces of the parameter space.<n>This allows us to build a fully-correlated approximate posterior reflecting the overparametrization.
  arXiv  Detail & Related papers  (2025-10-27T15:38:35Z)
• Abstain Mask Retain Core: Time Series Prediction by Adaptive Masking Loss with Representation Consistency [4.047219770183742]
  Time series forecasting plays a pivotal role in critical domains such as energy management and financial markets.<n>This study reveals a counterintuitive phenomenon: appropriately truncating historical data can enhance prediction accuracy.<n>We propose an innovative solution termed Adaptive Masking Loss with Representation Consistency.
  arXiv  Detail & Related papers  (2025-10-22T19:23:53Z)
• Towards the Training of Deeper Predictive Coding Neural Networks [44.14001498773255]
  Predictive coding networks are neural models that perform inference through an iterative energy minimization process.<n>While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers.<n>We show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers.
  arXiv  Detail & Related papers  (2025-06-30T12:44:47Z)
• Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learning [36.954726737451224]
  We consider the setting of a layer-peeled model in representation learning, which results in interesting phenomena related to learned features in deep learning and transfer learning. We show that DP fine-tuning is less robust compared to fine-tuning without DP, particularly in the presence of perturbations.
  arXiv  Detail & Related papers  (2024-05-14T19:18:19Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)
• Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets [57.06026574261203]
  We provide theoretical evidence for learning low-dimensional state spaces, which can also model long-term memory. Experiments corroborate our theory, demonstrating extrapolation via learning low-dimensional state spaces with both linear and non-linear RNNs.
  arXiv  Detail & Related papers  (2022-10-25T14:45:15Z)
• SGD with Large Step Sizes Learns Sparse Features [22.959258640051342]
  We showcase important features of the dynamics of the Gradient Descent (SGD) in the training of neural networks. We show that the longer large step sizes keep SGD high in the loss landscape, the better the implicit regularization can operate and find sparse representations.
  arXiv  Detail & Related papers  (2022-10-11T11:00:04Z)
• Transfer RL across Observation Feature Spaces via Model-Based Regularization [9.660642248872973]
  In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations. We propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task. Our algorithm works for drastic changes of observation space without any inter-task mapping or any prior knowledge of the target task.
  arXiv  Detail & Related papers  (2022-01-01T22:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.