WEBSERV: A Browser-Server Environment for Efficient Training of Reinforcement Learning-based Web Agents at Scale

• URL: http://arxiv.org/abs/2510.16252v1
• Date: Fri, 17 Oct 2025 22:54:33 GMT
• Title: WEBSERV: A Browser-Server Environment for Efficient Training of Reinforcement Learning-based Web Agents at Scale
• Abstract summary: Training and evaluation of Reinforcement Learning (RL) web agents have gained increasing attention.<n>Existing environments overwhelm policy models with excessive and noisy context.<n>We propose WEBSERV, a site-agnostic browser environment that balances context and action complexity.
• Score: 24.98872156421692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training and evaluation of Reinforcement Learning (RL) web agents have gained increasing attention, yet a scalable and efficient environment that couples realistic and robust browser-side interaction with controllable server-side state at scale is still missing. Existing environments tend to have one or more of the following issues: they overwhelm policy models with excessive and noisy context; they perform actions non-deterministically without waiting for the UI or network to stabilize; or they cannot scale isolated client-server containers effectively for parallel RL rollouts. We propose WEBSERV, an environment that includes 1) a compact, site-agnostic browser environment that balances context and action complexity, and 2) a scalable RL environment via efficient launching and resetting web-servers to enable scalable RL training and evaluation. We evaluate WEBSERV on the shopping CMS and Gitlab tasks in WebArena, achieving state-of-the-art single-prompt success rates while cutting launch latency by ~5x and storage need by ~240x, with a comparable memory footprint, enabling 200+ concurrent containers on a single host.

Related papers

• BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes [54.96895469970198]
  BrowserForge generates web interaction data at scale by driving many browser sandboxes in parallel over the open web.<n> BrowserForge couples three components: an open-web sourcing stage that exposes the agent to hundreds of thousands of real, openly reachable websites.<n>A rule-plus-model cleaning pipeline removes failed runs and rewrites the surviving reasoning into a single unified chain-of-thought style.
  arXiv  Detail & Related papers  (2026-08-25T17:35:42Z)
• MCP-Universe RL: A Framework for Training MCP Tool-Use Agents via Reinforcement Learning [54.082979654934]
  Reinforcement learning (RL) has become an effective way to improve the tool-use ability of large language models (LLMs)<n>Most existing RL frameworks stop at the policy update.<n>We present MCP-Universe RL (MCP-U RL), an open-source framework that takes over both.
  arXiv  Detail & Related papers  (2026-08-23T01:49:04Z)
• SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL [38.05711358650287]
  Computer use agents (CUAs) are emerging as a powerful interface for automating complex digital through visual perception and GUI execution.<n>Online reinforcement learning with verifiable rewards (RLVR) has emerged as a key direction for scaling their capabilities.<n>We introduce ScaleCUA, a unified framework that scales online RL for CUAs via verifiable task synthesis and efficient training.
  arXiv  Detail & Related papers  (2026-07-13T07:32:35Z)
• OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents [61.42551993396796]
  We introduce OpenWebRL, an open framework for training visual web agents with online multi-turn RL on real websites.<n>OpenWebRL covers the full training pipeline, including scalable live-browser infrastructure.<n>We train OpenWebRL-4B, which establishes a new open-source state of the art on challenging live-web benchmarks.
  arXiv  Detail & Related papers  (2026-06-01T10:20:10Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• Weblica: Scalable and Reproducible Training Environments for Visual Web Agents [5.132610127852326]
  Weblica is a framework for constructing reproducible and scalable web environments.<n>We scale RL training to thousands of diverse environments and tasks.<n>Our best model, Weblica-8B, outperforms open-weight baselines of similar size across multiple web navigation benchmarks.
  arXiv  Detail & Related papers  (2026-05-07T17:17:10Z)
• Web-Gewu: A Browser-Based Interactive Playground for Robot Reinforcement Learning [3.337549990679896]
  Web-Gewu is an interactive robotics education platform built on a cloud-edge-client collaborative architecture.<n>The system offloads all physics simulation and reinforcement learning training to the edge node.<n>Web-Gewu provides a highly scalable, out-of-the-box, and barrier-free teaching infrastructure for embodied intelligence.
  arXiv  Detail & Related papers  (2026-04-18T16:08:48Z)
• Safe and Scalable Web Agent Learning via Recreated Websites [27.277983097111335]
  Training web agents is fundamentally limited by environments they learn from.<n>We propose a framework that treats language models as environment creators, automatically cloning real-world websites into verifiable synthetic environments.
  arXiv  Detail & Related papers  (2026-03-11T07:58:53Z)
• SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents [44.0779548943909]
  Reinforcement learning (RL) has become a key paradigm for training software engineering (SWE) agents.<n>Existing pipelines typically rely on per-task containers for isolation.<n>We propose SWE-MiniSandbox, a container-free method that enables scalable RL training of SWE agents without sacrificing isolation.
  arXiv  Detail & Related papers  (2026-02-11T02:33:04Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• SWE-World: Building Software Engineering Agents in Docker-Free Environments [91.17484806743641]
  SWE-World is a Docker-free framework that replaces physical execution environments with a learned surrogate for training and evaluating software engineering agents.<n>We show that SWE-World raises Qwen2.5-Coder-32B from 6.2% to 52.0% via Docker-free SFT, 55.0% with Docker-free RL, and 68.2% with further TTS.
  arXiv  Detail & Related papers  (2026-02-03T11:44:39Z)
• WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks [35.99528846296261]
  WebGym is the largest-to-date open-source environment for training realistic visual web agents.<n>WebGym contains nearly 300,000 tasks with rubric-based evaluations across diverse, real-world websites.
  arXiv  Detail & Related papers  (2026-01-05T09:35:11Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video? [56.33950760097989]
  IWR-Bench is a novel benchmark for evaluating the capabilities of Large Vision-Language Models (LVLMs) in interactive webpage reconstruction from video.<n>IWR-Bench comprises 113 meticulously curated tasks from 100 real-world websites, with 1,001 actions.<n>This benchmark evaluates models on two fundamental challenges: comprehensive multi-modal reasoning to infer interaction logic from video and assets, and advanced code generation to translate this logic into functional code.
  arXiv  Detail & Related papers  (2025-09-29T12:38:06Z)
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning [78.86567400365392]
  We present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories.<n>To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation.<n>Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T03:24:08Z)
• ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents [40.033250603445246]
  We introduce ComputerRL, a framework for autonomous desktop intelligence.<n>ComputerRL features the API-GUI paradigm, which unifies programmatic API calls and direct GUI interaction.<n>We propose Entropulse, a training strategy that alternates reinforcement learning with supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-08-19T17:59:45Z)
• InternBootcamp Technical Report: Boosting LLM Reasoning with Verifiable Task Scaling [71.37579508777843]
  Large language models (LLMs) have revolutionized artificial intelligence by enabling complex reasoning capabilities.<n>To address this gap, we present InternBootcamp, an open-source framework comprising 1000+ domain-diverse task environments.
  arXiv  Detail & Related papers  (2025-08-12T05:00:00Z)
• RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems [2.4762227354811293]
  RecoMind is a simulator-based reinforcement learning framework designed for the effective optimization of session-based goals at web-scale.<n>We show that RecoMind significantly outperforms traditional supervised learning recommendation approaches in in-session user satisfaction.
  arXiv  Detail & Related papers  (2025-07-31T23:01:14Z)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites.<n>We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions.<n>Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)
• Steward: Natural Language Web Automation [19.301371856154965]
  Large language models (LLMs) have demonstrated exceptional capabilities in serving as the foundation for AI assistants. We introduce Steward, a novel LLM-powered web automation tool designed to serve as a cost-effective, scalable, end-to-end solution for automating web interactions. We discuss various design and implementation challenges, including state representation, action sequence selection, system responsiveness, detecting task completion, and caching implementation.
  arXiv  Detail & Related papers  (2024-09-23T18:06:32Z)
• NAVIX: Scaling MiniGrid Environments with JAX [17.944645332888335]
  We introduce NAVIX, a re-implementation of MiniGrid in JAX. NAVIX achieves over 200 000x speed improvements in batch mode, supporting up to 2048 agents in parallel on a single Nvidia A100 80 GB. This reduces experiment times from one week to 15 minutes, promoting faster design and more scalable RL model development.
  arXiv  Detail & Related papers  (2024-07-28T04:39:18Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.