Multivariate Distributional Reinforcement Learning Using Sliced Divergences

• URL: http://arxiv.org/abs/2605.31222v1
• Date: Fri, 29 May 2026 12:26:45 GMT
• Title: Multivariate Distributional Reinforcement Learning Using Sliced Divergences
• Abstract summary: Sliced Distributional Reinforcement Learning lifts tractable one-dimensional divergences to multivariate return distributions via projections.<n>We analyze Wasserstein, Cramér, and Maximum Mean Discrepancy (MMD)<n>We evaluate SDRL on a toy chain problem and a gridworld image-based environment as well as a subset of Atari games.
• Score: 37.5115685757579
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distributional reinforcement learning (DRL) models the full return distribution rather than expectations, but extending it to multivariate settings remains challenging. Many common metrics do not naturally generalize beyond one dimension or lose computational tractability, and the multivariate case introduces additional difficulties such as general matrix discounting, for which no contraction results are available. We introduce Sliced Distributional Reinforcement Learning (SDRL), which lifts tractable one-dimensional divergences to multivariate return distributions via projections. We prove Bellman contraction for uniform slicing under shared scalar discounting, and introduce a maximum-slicing variant with contraction under general dense discount matrices. SDRL supports a broad class of base divergences; we analyze Wasserstein, Cramér, and Maximum Mean Discrepancy (MMD), and characterize which SDRL variants suit the standard single-sample Bellman update used in distributional RL. We evaluate SDRL on a toy chain problem and a gridworld image-based environment as well as a subset of Atari games.

Related papers

• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• Calibrated Multimodal Representation Learning with Missing Modalities [100.55774771852468]
  Multimodal representation learning harmonizes distinct modalities by aligning them into a unified latent space.<n>Recent research generalizes traditional cross-modal alignment to produce enhanced multimodal synergy but requires all modalities to be present for a common instance.<n>We provide theoretical insights into this issue from an anchor shift perspective.<n>We propose CalMRL for multimodal representation learning to calibrate incomplete alignments caused by missing modalities.
  arXiv  Detail & Related papers  (2025-11-15T05:01:43Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• Distribution-dependent Generalization Bounds for Tuning Linear Regression Across Tasks [24.2043855572415]
  We find distribution-dependent bounds on the generalization error for the validation loss when tuning the L1 and L2 coefficients.<n>We extend our results to a generalization of ridge regression, where we achieve tighter bounds that take into account the mean of the ground truth distribution.
  arXiv  Detail & Related papers  (2025-07-07T15:08:45Z)
• Bayesian Multivariate Density-Density Regression [25.35298354797079]
  We introduce a novel and scalable Bayesian framework for multivariate-density-density regression (DDR)<n>Our approach addresses the critical issue of distributions residing in spaces of differing dimensions.<n>We show that Bayesian DDR provides robust fits, superior predictive performance compared to traditional methods, and valuable insights into complex biological interactions.
  arXiv  Detail & Related papers  (2025-04-17T03:46:03Z)
• Diverse Projection Ensembles for Distributional Reinforcement Learning [6.144680854063937]
  distributional reinforcement learning algorithms aim to learn the distribution of returns rather than their expected value.<n>We study the combination of several different projections and representations in a distributional ensemble.<n>We derive an algorithm that uses ensemble disagreement, measured by the average 1-Wasserstein distance, as a bonus for deep exploration.
  arXiv  Detail & Related papers  (2023-06-12T13:59:48Z)
• Policy Evaluation in Distributional LQR [70.63903506291383]
  We provide a closed-form expression of the distribution of the random return. We show that this distribution can be approximated by a finite number of random variables. Using the approximate return distribution, we propose a zeroth-order policy gradient algorithm for risk-averse LQR.
  arXiv  Detail & Related papers  (2023-03-23T20:27:40Z)
• The Generalization Error of Stochastic Mirror Descent on Over-Parametrized Linear Models [37.6314945221565]
  Deep networks are known to generalize well to unseen data. Regularization properties ensure interpolating solutions with "good" properties are found. We present simulation results that validate the theory and introduce two data models.
  arXiv  Detail & Related papers  (2023-02-18T22:23:42Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.