SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning

• URL: http://arxiv.org/abs/2604.22558v1
• Date: Fri, 24 Apr 2026 13:53:39 GMT
• Title: SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning
• Abstract summary: Reinforcement Learning (RL) has emerged as a promising paradigm for training MLLM agents on dynamic GUI tasks.<n>Online RL captures the long-term dynamics but suffers from high interaction costs and potential environmental instability.<n>We propose SOLAR-RL (Semi-Online Long-horizon Assignment Reinforcement Learning), which integrates global trajectory insights directly into the offline learning process.
• Score: 21.3755929369092
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As Multimodal Large Language Models (MLLMs) mature, GUI agents are evolving from static interactions to complex navigation. While Reinforcement Learning (RL) has emerged as a promising paradigm for training MLLM agents on dynamic GUI tasks, its effective application faces a dilemma. Standard Offline RL often relies on static step-level data, neglecting global trajectory semantics such as task completion and execution quality. Conversely, Online RL captures the long-term dynamics but suffers from high interaction costs and potential environmental instability. To bridge this gap, we propose SOLAR-RL (Semi-Online Long-horizon Assignment Reinforcement Learning). Instead of relying solely on expensive online interactions, our framework integrates global trajectory insights directly into the offline learning process. Specifically, we reconstruct diverse rollout candidates from static data, detect the first failure point using per-step validity signals, and retroactively assign dense step-level rewards with target-aligned shaping to reflect trajectory-level execution quality, effectively simulating online feedback without interaction costs. Extensive experiments demonstrate that SOLAR-RL significantly improves long-horizon task completion rates and robustness compared to strong baselines, offering a sample-efficient solution for autonomous GUI navigation.

Related papers

• UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents [50.053654092780825]
  Online Reinforcement Learning (RL) offers a promising paradigm for enhancing GUI agents through direct environment interaction.<n>We propose UI-Mem, a novel framework that enhances GUI online RL with a Hierarchical Experience Memory.<n>We show that UI-Mem significantly outperforms traditional RL baselines and static reuse strategies.
  arXiv  Detail & Related papers  (2026-02-05T16:21:43Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation [65.3648667980258]
  Vision-language model (VLM) based GUI agents show promise for automating complex tasks, but face significant challenges in applying reinforcement learning (RL)<n>We propose DART, a Decoupled Agentic RL Training framework for GUI agents, which coordinates heterogeneous modules in a highly decoupled manner.<n>On the OSWorld benchmark, DART-GUI-7B achieves a 42.13% task success rate, a 14.61% absolute gain over the base model, and 7.34% higher than open-source SOTA.
  arXiv  Detail & Related papers  (2025-09-28T13:19:20Z)
• Multi-Agent Path Finding via Offline RL and LLM Collaboration [0.0]
  Multi-Agent Path Finding (MAPF) poses a significant and challenging problem for applications in robotics and logistics.<n>We propose an efficient decentralized planning framework based on the Decision Transformer (DT)<n>Our approach effectively handles long-horizon credit assignment and significantly improves performance in scenarios with sparse and delayed rewards.
  arXiv  Detail & Related papers  (2025-09-26T09:53:40Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning [78.86567400365392]
  We present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories.<n>To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation.<n>Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T03:24:08Z)
• Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning [33.48496141312585]
  Reinforcement learning (RL) has been widely adopted for controlling and optimizing complex engineering systems such as next-generation wireless networks. An important challenge in adopting RL is the need for direct access to the physical environment. We propose an offline MARL scheme that integrates distributional RL and conservative Q-learning to address the environment's inherent aleatoric uncertainty.
  arXiv  Detail & Related papers  (2024-02-13T12:49:22Z)
• Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization [23.416448404647305]
  OMIGA is a new offline m ulti-agent RL algorithm with implicit global-to-local v alue regularization. We show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.
  arXiv  Detail & Related papers  (2023-07-21T14:37:54Z)
• ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles [52.34951901588738]
  We propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods.
  arXiv  Detail & Related papers  (2023-06-12T05:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.