Offline Reinforcement Learning with Universal Horizon Models

• URL: http://arxiv.org/abs/2605.15603v1
• Date: Fri, 15 May 2026 04:30:30 GMT
• Title: Offline Reinforcement Learning with Universal Horizon Models
• Abstract summary: We introduce universal horizon models (UHM), a generalization of GHM that directly predicts future states under arbitrary horizons.<n>We propose a scalable value learning method that employs a winsorized horizon distribution to stabilize training by capping excessively large horizons.<n> Experimental results on 100 challenging OGBench tasks demonstrate that the proposed method outperforms competitive baselines.
• Score: 15.801661284310823
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model-based reinforcement learning (RL) offers a compelling approach to offline RL by enabling value learning on imagined on-policy trajectories. However, it often suffers from compounding errors due to repeated model inference on self-generated states. While geometric horizon models (GHM) alleviate this issue through direct prediction over a discounted infinite-horizon future, they remain challenged in accurately modeling distant future states. To this end, we introduce universal horizon models (UHM), a generalization of GHM that directly predicts future states under arbitrary horizons. Leveraging this flexibility, we propose a scalable value learning method that employs a winsorized horizon distribution to stabilize training by capping excessively large horizons. Experimental results on 100 challenging OGBench tasks demonstrate that the proposed method outperforms competitive baselines, particularly on tasks with highly suboptimal datasets and those requiring long-horizon reasoning. Project page: https://rllab-snu.github.io/projects/UHM/

Related papers

• AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution [61.593935260052795]
  Post-training viaSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is crucial for enhancing reasoning in Multimodal Large Language Models (MLLMs)<n>Existing paradigms often reach a performance bottleneck due to the limitations of static data.<n>We propose Anchor Evolution (AnE), a new paradigm that integrates truth-anchored data curation and model evolution.
  arXiv  Detail & Related papers  (2026-05-25T08:26:34Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• Horizon Imagination: Efficient On-Policy Rollout in Diffusion World Models [53.30870288484743]
  We study diffusion-based world models for reinforcement learning.<n>Current methods either require heavyweight models at inference or rely on highly sequential imagination.<n>We propose Horizon Imagination (HI), an on-policy imagination process for discrete policies that denoises multiple future observations in parallel.
  arXiv  Detail & Related papers  (2026-02-08T16:07:04Z)
• LIVE: Long-horizon Interactive Video World Modeling [39.52605866460851]
  Long-horizon Interactive Video world modEl enforces bounded error accumulation via a novel cycle-consistency objective.<n>Live achieves state-of-the-art performance on long-horizon benchmarks, generating stable, high-quality videos far beyond training rollout lengths.
  arXiv  Detail & Related papers  (2026-02-03T17:10:03Z)
• Imitation from Observations with Trajectory-Level Generative Embeddings [18.63253047959276]
  We consider the offline imitation learning from observations (LfO) where the expert demonstrations are scarce and the available offline suboptimal data are far from the expert behavior.<n>We propose TGE, a trajectory-level generative embedding for offline LfO that constructs a dense, smooth surrogate reward by estimating expert state density in the latent space of a temporal diffusion model trained on offline trajectory data.
  arXiv  Detail & Related papers  (2026-01-01T19:38:37Z)
• Visual Autoregressive Modelling for Monocular Depth Estimation [69.01449528371916]
  We propose a monocular depth estimation method based on visual autoregressive ( VAR) priors.<n>Our method adapts a large-scale text-to-image VAR model and introduces a scale-wise conditional upsampling mechanism.<n>We report state-of-the-art performance in indoor benchmarks under constrained training conditions, and strong performance when applied to outdoor datasets.
  arXiv  Detail & Related papers  (2025-12-27T17:08:03Z)
• Generative Actor Critic [74.04971271003869]
  Generative Actor Critic (GAC) is a novel framework that decouples sequential decision-making by reframing textitpolicy evaluation as learning a generative model of the joint distribution over trajectories and returns.<n>Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-12-25T06:31:11Z)
• Scalable Offline Model-Based RL with Action Chunks [60.80151356018376]
  We study whether model-based reinforcement learning can provide a scalable recipe for tackling complex, long-horizon tasks in offline RL.<n>We call this recipe textbfModel-Based RL with Action Chunks (MAC).<n>We show that MAC achieves the best performance among offline model-based RL algorithms, especially on challenging long-horizon tasks.
  arXiv  Detail & Related papers  (2025-12-08T23:26:29Z)
• RAD: Retrieval High-quality Demonstrations to Enhance Decision-making [23.136426643341462]
  offline reinforcement learning (RL) enables agents to learn policies from fixed datasets.<n>RL is often limited by dataset sparsity and the lack of transition overlap between suboptimal and expert trajectories.<n>We propose Retrieval High-quAlity Demonstrations (RAD) for decision-making, which combines non-parametric retrieval with diffusion-based generative modeling.
  arXiv  Detail & Related papers  (2025-07-21T08:08:18Z)
• Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning [6.345851712811528]
  We introduce a novel model-based offline RL method, Lower Expectile Q-learning (LEQ)<n>LEQ provides a low-bias model-based value estimation via lower expectile regression of $lambda$-returns.<n>Our studies demonstrate that lower expectile regression, $lambda$-returns, and critic training on offline data are all crucial for LEQ.
  arXiv  Detail & Related papers  (2024-06-30T13:44:59Z)
• Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning [35.5986167075913]
  Reinforcement Learning (RL) methods are typically applied directly in environments to learn policies. We aim to build a simple and discrete world model that abstracts the original environment. Our experiments on the D4RL benchmark show that VMG can outperform state-of-the-art offline RL methods in several goal-oriented tasks.
  arXiv  Detail & Related papers  (2022-06-09T09:51:42Z)
• Offline Reinforcement Learning from Images with Latent Space Models [60.69745540036375]
  offline reinforcement learning (RL) refers to the problem of learning policies from a static dataset of environment interactions. We build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces. Our approach is both tractable in practice and corresponds to maximizing a lower bound of the ELBO in the unknown POMDP.
  arXiv  Detail & Related papers  (2020-12-21T18:28:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.