Structured Distillation of Web Agent Capabilities Enables Generalization

• URL: http://arxiv.org/abs/2604.07776v1
• Date: Thu, 09 Apr 2026 04:04:15 GMT
• Title: Structured Distillation of Web Agent Capabilities Enables Generalization
• Abstract summary: We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles.<n>Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments.<n>The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol.
• Score: 33.41723014134473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io

Related papers

• Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent [98.33217711280383]
  We introduce Agents-A1, a 35B Mixture-of-Experts Agentic Model that reaches trillion- parameter-level performance by scaling the agent horizon.<n>We build a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes.<n>Agents-A1 achieves strong and broad performance for long-horizon agent benchmarks.
  arXiv  Detail & Related papers  (2026-06-29T17:50:54Z)
• What Makes Interaction Trajectories Effective for Training Terminal Agents? [55.62817294510983]
  Terminal-Lego is a scalable pipeline that transforms real-world issues into environment-verified agentic tasks.<n>Students fine-tuned on trajectories from DeepSeek-V3.2, a lower-scoring agent, exhibit significantly stronger generalization.<n>With only 15.3k Terminal-Lego trajectories, Qwen3-32B achieves a 24.3% score on Terminal-Bench 2.0, rivaling previous SOTA performance established with over 30x the data volume.
  arXiv  Detail & Related papers  (2026-06-02T10:37:47Z)
• Orchard: An Open-Source Agentic Modeling Framework [124.68499958175111]
  We present Orchard, an open-source framework for scalable agentic modeling.<n>At its core is Orchard Env, a lightweight environment service providing reusable primitives for sandbox lifecycle management.<n>On top of Orchard Env, we build three agentic modeling recipes.
  arXiv  Detail & Related papers  (2026-05-14T16:35:12Z)
• ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation [16.04915509466256]
  ClassEval-Pro is a benchmark of 300 class-level tasks spanning 11 domains.<n>The best model achieves only 45.6% class-level Pass@1, with a 17.7-point gap between the strongest and weakest models.
  arXiv  Detail & Related papers  (2026-04-29T17:38:37Z)
• From Imitation to Discrimination: Progressive Curriculum Learning for Robust Web Navigation [18.965820679356366]
  We introduce the Triton dataset (590k instances) and a progressive training curriculum.<n>We produce three models: Triton-SFT-32B for basic imitation, Triton-ORPO-32B for robust discrimination, and Triton-GRPO-32B for long-horizon consistency.<n> Empirical evaluation on Mind2Web demonstrates that Triton-GRPO-32B achieves state-of-the-art performance among open-source models.
  arXiv  Detail & Related papers  (2026-04-14T12:37:45Z)
• Yet Even Less Is Even Better For Agentic, Reasoning, and Coding LLMs [28.672417618778976]
  Training effective software engineering agents requires large volumes of task-specific trajectories.<n>We propose an end-to-end training framework that achieves superior agentic capabilities with fewer but higher-quality training trajectories.
  arXiv  Detail & Related papers  (2026-04-01T12:33:25Z)
• EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environments [0.10934862523101825]
  We show that training AI agents on high-fidelity reinforcement learning environments produces capabilities that generalize beyond the training distribution.<n>We introduce CoreCraft, the first environment in EnterpriseBench, Surge AI's suite of agentic RL environments.
  arXiv  Detail & Related papers  (2026-02-18T04:35:46Z)
• Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters [169.7981969517903]
  Step 3.5 Flash bridges frontier-level agentic intelligence and computational efficiency.<n>We focus on what matters most when building agents: sharp reasoning and fast, reliable execution.
  arXiv  Detail & Related papers  (2026-02-11T07:53:51Z)
• Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem [90.17610617854247]
  We introduce the Agentic Learning Ecosystem (ALE), a foundational infrastructure that optimize the production pipeline for agentic model.<n>ALE consists of three components: ROLL, a post-training framework for weight optimization; ROCK, a sandbox environment manager for trajectory generation; and iFlow CLI, an agent framework for efficient context engineering.<n>We release ROME, an open-source agent grounded by ALE and trained on over one million trajectories.
  arXiv  Detail & Related papers  (2025-12-31T14:03:39Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training [73.90260246781435]
  We present Lory, the first approach that scales such architectures to autoregressive language model pre-training. We show significant performance gains over parameter-matched dense models on both perplexity and a variety of downstream tasks. Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing.
  arXiv  Detail & Related papers  (2024-05-06T03:06:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.