ClawEnvKit: Automatic Environment Generation for Claw-Like Agents

• URL: http://arxiv.org/abs/2604.18543v3
• Date: Wed, 29 Apr 2026 14:01:51 GMT
• Title: ClawEnvKit: Automatic Environment Generation for Claw-Like Agents
• Abstract summary: We introduce ClawEnvKit, an autonomous generation pipeline capable of generating verified environments on demand.<n>ClawEnvKit comprises three modules: (1) a pipeline that extracts structured generation parameters from natural language input; (2) a generator that produces the task specification, tool interface, and scoring configuration; and (3) a validator that enforces feasibility, diversity, structural validity, and internal consistency.<n>We construct Auto-ClawEval, the first large-scale benchmark for claw-like agents, comprising 1,040 environments across 24 categories.
• Score: 85.29126619772153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Constructing environments for training and evaluating claw-like agents remains a manual, human-intensive process that does not scale. We argue that what is needed is not just a dataset, but an automated pipeline capable of generating diverse, verified environments on demand. To this end, we introduce ClawEnvKit, an autonomous generation pipeline that instantiates this formalism from natural language descriptions. The pipeline comprises three modules: (1) a parser that extracts structured generation parameters from natural language input; (2) a generator that produces the task specification, tool interface, and scoring configuration; and (3) a validator that enforces feasibility, diversity, structural validity, and internal consistency across the generated environments. Using ClawEnvKit, we construct Auto-ClawEval, the first large-scale benchmark for claw-like agents, comprising 1,040 environments across 24 categories. Empirically, Auto-ClawEval matches or exceeds human-curated environments on coherence and clarity at 13,800x lower cost. Evaluated across 4 model families and 8 agent harness frameworks, we find that harness engineering boosts performance by up to 15.7 percentage points over a bare ReAct baseline, completion remains the primary axis of variation with no model saturating the benchmark, and automated generation enables evaluation at a scale previously infeasible. Beyond static benchmarking, ClawEnvKit enables live evaluation: users describe a desired capability in natural language and obtain a verified environment on demand, turning evaluation into a continuous, user-driven process. The same mechanism serves as an on-demand training environment generator, producing task distributions that adapt to an agent's current weaknesses rather than being bounded by existing user logs.

Related papers

• EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agent [21.572853994120806]
  EnvCraft is an automated framework for synthesizing executable environments and scalable training data.<n>We synthesize 139 interactive environments comprising approximately 20K complex tasks for Agentic RL training.
  arXiv  Detail & Related papers  (2026-09-04T10:13:41Z)
• Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalization [93.59040719904154]
  This paper introduces RACES (textbfRecursive textbfAutomated textbfComposition for textbfEnvironment textbfScaling), a framework that conceptualizes verifiable environments as composable building blocks.<n> RACES is implemented with 300 individual environments and defines a set of composition operators that induce diverse reasoning patterns.
  arXiv  Detail & Related papers  (2026-06-10T17:39:41Z)
• EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL [54.09410318521061]
  We introduce EnvFactory, a fully automated framework for Agentic Reinforcement Learning (Agentic RL) training.<n>EnvFactory autonomously explores and stateful, executable tool environments from authentic resources.<n>It synthesizes natural multi-turn trajectories through topology-aware sampling and calibrated refinement.<n>It achieves superior training efficiency and downstream performance, improving Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks including $2$-Bench and VitaBench.
  arXiv  Detail & Related papers  (2026-05-18T17:37:40Z)
• Computer Use at the Edge of the Statistical Precipice [19.604476051265852]
  We show that a 1MB replay script that blindly executes a recorded action sequence outperforms frontier models on prominent static benchmarks.<n>We trace failures to two root causes: non-principled environment design and non-principled evaluation methodology.
  arXiv  Detail & Related papers  (2026-05-07T22:43:43Z)
• ClawGym: A Scalable Framework for Building Effective Claw Agents [47.47444724291439]
  We present ClawGym, a scalable framework that supports the full lifecycle of Claw-style personal agent development.<n>ClawGym-SynData is a dataset of 13.5K filtered tasks synthesized from persona-driven intents and skill-grounded operations.<n>We then train a family of capable Claw-style models, termed ClawGym-Agents, through supervised fine-tuning on black-box rollout trajectories.<n>We further construct ClawGym-Bench, a benchmark of 200 instances through automated filtering and human-LLM review.
  arXiv  Detail & Related papers  (2026-04-29T17:12:22Z)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning [62.499592503950026]
  Large language model (LLM) have empowered autonomous agents to perform complex tasks that require multi-turn interactions with tools and environments.<n>We propose Agent World Model (AWM), a fully synthetic environment generation pipeline.<n>We scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets.
  arXiv  Detail & Related papers  (2026-02-10T18:55:41Z)
• AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning [71.4322853508083]
  Conducting reinforcement learning in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents.<n>Previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth.<n>We propose a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks.
  arXiv  Detail & Related papers  (2025-12-28T09:43:11Z)
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments [70.42705564227548]
  We propose an automated environment construction pipeline for large language models (LLMs)<n>This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools.<n>We also introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution.
  arXiv  Detail & Related papers  (2025-08-12T09:45:19Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data [26.004807291215258]
  Language-conditioned robot manipulation aims to develop robots capable of understanding and executing complex tasks. We propose a general-purpose, language-conditioned approach that combines base skill priors and imitation learning under unstructured data. We assess our model's performance in both simulated and real-world environments using a zero-shot setting.
  arXiv  Detail & Related papers  (2023-05-30T14:40:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.