WEBSERV: A Browser-Server Environment for Efficient Training of Reinforcement Learning-based Web Agents at Scale

• URL: http://arxiv.org/abs/2510.16252v1
• Date: Fri, 17 Oct 2025 22:54:33 GMT
• Title: WEBSERV: A Browser-Server Environment for Efficient Training of Reinforcement Learning-based Web Agents at Scale
• Authors: Yuxuan Lu, Jing Huang, Hui Liu, Jiri Gesi, Yan Han, Shihan Fu, Tianqi Zheng, Dakuo Wang,
• Abstract summary: Training and evaluation of Reinforcement Learning (RL) web agents have gained increasing attention.<n>Existing environments overwhelm policy models with excessive and noisy context.<n>We propose WEBSERV, a site-agnostic browser environment that balances context and action complexity.
• Score: 24.98872156421692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training and evaluation of Reinforcement Learning (RL) web agents have gained increasing attention, yet a scalable and efficient environment that couples realistic and robust browser-side interaction with controllable server-side state at scale is still missing. Existing environments tend to have one or more of the following issues: they overwhelm policy models with excessive and noisy context; they perform actions non-deterministically without waiting for the UI or network to stabilize; or they cannot scale isolated client-server containers effectively for parallel RL rollouts. We propose WEBSERV, an environment that includes 1) a compact, site-agnostic browser environment that balances context and action complexity, and 2) a scalable RL environment via efficient launching and resetting web-servers to enable scalable RL training and evaluation. We evaluate WEBSERV on the shopping CMS and Gitlab tasks in WebArena, achieving state-of-the-art single-prompt success rates while cutting launch latency by ~5x and storage need by ~240x, with a comparable memory footprint, enabling 200+ concurrent containers on a single host.

Related papers

• SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents [44.0779548943909]
  Reinforcement learning (RL) has become a key paradigm for training software engineering (SWE) agents.<n>Existing pipelines typically rely on per-task containers for isolation.<n>We propose SWE-MiniSandbox, a container-free method that enables scalable RL training of SWE agents without sacrificing isolation.
  arXiv  Detail & Related papers  (2026-02-11T02:33:04Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• SWE-World: Building Software Engineering Agents in Docker-Free Environments [91.17484806743641]
  SWE-World is a Docker-free framework that replaces physical execution environments with a learned surrogate for training and evaluating software engineering agents.<n>We show that SWE-World raises Qwen2.5-Coder-32B from 6.2% to 52.0% via Docker-free SFT, 55.0% with Docker-free RL, and 68.2% with further TTS.
  arXiv  Detail & Related papers  (2026-02-03T11:44:39Z)
• WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks [35.99528846296261]
  WebGym is the largest-to-date open-source environment for training realistic visual web agents.<n>WebGym contains nearly 300,000 tasks with rubric-based evaluations across diverse, real-world websites.
  arXiv  Detail & Related papers  (2026-01-05T09:35:11Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video? [56.33950760097989]
  IWR-Bench is a novel benchmark for evaluating the capabilities of Large Vision-Language Models (LVLMs) in interactive webpage reconstruction from video.<n>IWR-Bench comprises 113 meticulously curated tasks from 100 real-world websites, with 1,001 actions.<n>This benchmark evaluates models on two fundamental challenges: comprehensive multi-modal reasoning to infer interaction logic from video and assets, and advanced code generation to translate this logic into functional code.
  arXiv  Detail & Related papers  (2025-09-29T12:38:06Z)
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning [78.86567400365392]
  We present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories.<n>To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation.<n>Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T03:24:08Z)
• ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents [40.033250603445246]
  We introduce ComputerRL, a framework for autonomous desktop intelligence.<n>ComputerRL features the API-GUI paradigm, which unifies programmatic API calls and direct GUI interaction.<n>We propose Entropulse, a training strategy that alternates reinforcement learning with supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-08-19T17:59:45Z)
• InternBootcamp Technical Report: Boosting LLM Reasoning with Verifiable Task Scaling [71.37579508777843]
  Large language models (LLMs) have revolutionized artificial intelligence by enabling complex reasoning capabilities.<n>To address this gap, we present InternBootcamp, an open-source framework comprising 1000+ domain-diverse task environments.
  arXiv  Detail & Related papers  (2025-08-12T05:00:00Z)
• RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems [2.4762227354811293]
  RecoMind is a simulator-based reinforcement learning framework designed for the effective optimization of session-based goals at web-scale.<n>We show that RecoMind significantly outperforms traditional supervised learning recommendation approaches in in-session user satisfaction.
  arXiv  Detail & Related papers  (2025-07-31T23:01:14Z)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites.<n>We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions.<n>Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)
• Steward: Natural Language Web Automation [19.301371856154965]
  Large language models (LLMs) have demonstrated exceptional capabilities in serving as the foundation for AI assistants. We introduce Steward, a novel LLM-powered web automation tool designed to serve as a cost-effective, scalable, end-to-end solution for automating web interactions. We discuss various design and implementation challenges, including state representation, action sequence selection, system responsiveness, detecting task completion, and caching implementation.
  arXiv  Detail & Related papers  (2024-09-23T18:06:32Z)
• NAVIX: Scaling MiniGrid Environments with JAX [17.944645332888335]
  We introduce NAVIX, a re-implementation of MiniGrid in JAX. NAVIX achieves over 200 000x speed improvements in batch mode, supporting up to 2048 agents in parallel on a single Nvidia A100 80 GB. This reduces experiment times from one week to 15 minutes, promoting faster design and more scalable RL model development.
  arXiv  Detail & Related papers  (2024-07-28T04:39:18Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.