Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning

• URL: http://arxiv.org/abs/2605.26012v1
• Date: Mon, 25 May 2026 16:31:33 GMT
• Title: Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning
• Abstract summary: We present a representation-level prior that inserts a fixed orthonormal projection to constrain encoder features to a low-dimensional subspace.<n>Under a linear realizability assumption, we prove that when the bottleneck dimension exceeds the intrinsic rank of the optimal value function in feature space, the bottleneck preserves expressivity.<n>We find that across both single and multi-task benchmarks, baseline performance is either matched or improved once the bottleneck dimension exceeds a small task-dependent threshold.
• Score: 47.218458651895766
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep reinforcement learning (RL) agents commonly rely on high-dimensional neural representations, despite growing evidence that task-relevant value and policy structure may be intrinsically low-dimensional. In this work, we present a simple yet effective representation-level prior that inserts a fixed orthonormal projection to constrain encoder features to a low-dimensional subspace, requiring no auxiliary objectives, pretraining, or changes to the underlying RL algorithm. Under a linear realizability assumption, we prove that when the bottleneck dimension exceeds the intrinsic rank of the optimal value function in feature space, the bottleneck preserves expressivity and leaves the induced gradient dynamics unchanged up to an equivalent low-dimensional parameterization. Empirically, we find that across both single and multi-task benchmarks, baseline performance is either matched or improved once the bottleneck dimension exceeds a small task-dependent threshold; in many cases, value representations can be compressed to extremely low dimensions without loss, and the minimal sufficient dimension depends far more on environment complexity than encoder width. In addition, we analyze representation geometry and find that orthogonal bottlenecks stabilize feature norms and are associated with higher effective rank. Together, these results support a representation-space interpretation of the manifold hypothesis in reinforcement learning and position orthogonal bottlenecks as a lightweight, architecture-agnostic mechanism for shaping RL representations.

Related papers

• Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model [23.79711176655533]
  We study a globally high-degree function represented by a combination of latent compositional features whose weights decrease as a power law.<n>We show that a layer-wise spectral algorithm adapted to this structure achieves improved scaling relative to shallow, non-adaptive methods.<n>We prove sharp feature-wise recovery thresholds and show that aggregating these transitions yields an explicit power-law decay of the prediction error.
  arXiv  Detail & Related papers  (2026-05-14T08:37:28Z)
• Low-Rank Plus Sparse Matrix Transfer Learning under Growing Representations and Ambient Dimensions [6.949116398973296]
  We study transfer learning for structured matrix estimation under simultaneous growth of the ambient dimension and the intrinsic representation.<n>We propose a general transfer framework in which the target parameter decomposes into an embedded source component, low-dimensional low-rank innovations, and sparse edits.<n>We establish deterministic error bounds that separate target noise, representation growth, and source estimation error, yielding strictly improved rates when rank and sparsity increments are small.
  arXiv  Detail & Related papers  (2026-01-29T15:40:05Z)
• Directly Constructing Low-Dimensional Solution Subspaces in Deep Neural Networks [0.0]
  This work provides a novel paradigm for the Trainspace Sub-Native Distillation.<n>By defining the negligible subspace directly, we provide a search for student degradation.
  arXiv  Detail & Related papers  (2025-12-29T12:13:15Z)
• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization [88.7827307535107]
  Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature.<n>Existing solver-based acceleration methods often face significant image quality degradation under a low-dimensional budget.<n>We propose the Ensemble Parallel Direction solver (dubbed as EPD-EPr), a novel ODE solver that mitigates these errors by incorporating multiple gradient parallel evaluations in each step.
  arXiv  Detail & Related papers  (2025-12-28T05:48:55Z)
• Least Volume Analysis [2.799971758656275]
  Least Volume is a simple yet effective regularization method inspired by geometric intuition.<n>We show that LV induces a PCA-like importance ordering in nonlinear models.<n>We extend LV to non-Euclidean settings as Generalized Least Volume (GLV), enabling the integration of label information into the latent representation.
  arXiv  Detail & Related papers  (2024-04-27T04:09:49Z)
• Subspace Modeling for Fast Out-Of-Distribution and Anomaly Detection [5.672132510411465]
  This paper presents a principled approach for detecting anomalous and out-of-distribution (OOD) samples in deep neural networks (DNN) We propose the application of linear statistical dimensionality reduction techniques on the semantic features produced by a DNN. We show that the "feature reconstruction error" (FRE), which is the $ell$-norm of the difference between the original feature in the high-dimensional space and the pre-image of its low-dimensional reduced embedding, is highly effective for OOD and anomaly detection.
  arXiv  Detail & Related papers  (2022-03-20T00:55:20Z)
• Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting [60.98700344526674]
  Low-complexity models such as linear function representation play a pivotal role in enabling sample-efficient reinforcement learning. In this paper, we investigate a new sampling protocol, which draws samples in an online/exploratory fashion but allows one to backtrack and revisit previous states in a controlled and infrequent manner. We develop an algorithm tailored to this setting, achieving a sample complexity that scales practicallyly with the feature dimension, the horizon, and the inverse sub-optimality gap, but not the size of the state/action space.
  arXiv  Detail & Related papers  (2021-05-17T17:22:07Z)
• FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs [53.710405006523274]
  This work focuses on the representation learning question: how can we learn such features? Under the assumption that the underlying (unknown) dynamics correspond to a low rank transition matrix, we show how the representation learning question is related to a particular non-linear matrix decomposition problem. We develop FLAMBE, which engages in exploration and representation learning for provably efficient RL in low rank transition models.
  arXiv  Detail & Related papers  (2020-06-18T19:11:18Z)
• MetaSDF: Meta-learning Signed Distance Functions [85.81290552559817]
  Generalizing across shapes with neural implicit representations amounts to learning priors over the respective function space. We formalize learning of a shape space as a meta-learning problem and leverage gradient-based meta-learning algorithms to solve this task.
  arXiv  Detail & Related papers  (2020-06-17T05:14:53Z)
• Deep Dimension Reduction for Supervised Representation Learning [51.10448064423656]
  We propose a deep dimension reduction approach to learning representations with essential characteristics. The proposed approach is a nonparametric generalization of the sufficient dimension reduction method. We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero.
  arXiv  Detail & Related papers  (2020-06-10T14:47:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.