LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks

• URL: http://arxiv.org/abs/2604.13072v1
• Date: Fri, 20 Mar 2026 16:08:21 GMT
• Title: LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks
• Abstract summary: We introduce LiveClawBench, a benchmark to evaluate LLM agents on real-world assistant tasks.<n>Based on an analysis of various realClaw usage cases, we derive a Triple-Axis Complexity Framework.<n>We construct a pilot benchmark with explicit complexity-factor annotations, covering real-world assistant tasks.
• Score: 58.3639630490749
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based agents are increasingly expected to handle real-world assistant tasks, yet existing benchmarks typically evaluate them under isolated sources of difficulty, such as a single environment or fully specified instructions. This leaves a substantial gap between current evaluation settings and the compositional challenges that arise in practical deployment. To address this gap, we introduce LiveClawBench, a benchmark to evaluate LLM agents on real-world assistant tasks. Based on an analysis of various real OpenClaw usage cases, we derive a Triple-Axis Complexity Framework that characterizes task difficulty along three dimensions: Environment Complexity, Cognitive Demand, and Runtime Adaptability. Guided by this framework, we construct a pilot benchmark with explicit complexity-factor annotations, covering real-world assistant tasks with compositional difficulty. Together, the framework and benchmark provide a principled foundation for evaluating LLM agents in realistic assistant settings, and establish a basis for future expansion across task domains and complexity axes. We are continuing to enrich our case collections to achieve more comprehensive domain and complexity coverage. The project page is at https://github.com/Mosi-AI/LiveClawBench.

Related papers

• Learning Simple Test-Time Environments for LLM Web Agents [25.206333684249987]
  We introduce trial steps for agents to decompose a complex environment observation into sub-modules.<n>We implement a label-free learning method, Test-Time Environment Decomposition (TTED), to adapt agent behaviors with experience during inference.
  arXiv  Detail & Related papers  (2026-08-29T14:45:09Z)
• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• Multi-agent Framework for Time-Sensitive Complementary Collaboration in Minecraft [23.22494669795205]
  TickingCollabBench is a Minecraft-based multi-agent benchmark for a novel class of time-sensitive complementary collaboration tasks.<n>We develop the TickingCollab framework, which supports the generation of diverse dynamic environments.<n>We design a feasibility-aware automated benchmark generation pipeline, where an LLM drafts structurally diverse task configurations.
  arXiv  Detail & Related papers  (2026-06-14T09:02:33Z)
• REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version) [32.69257865315466]
  Large Language Models (LLMs) have achieved remarkable progress in recent years, driving their adoption across a wide range of domains.<n>In reverse engineering, LLMs are increasingly applied to critical tasks such as function and variable name recovery and type inference.<n>Existing studies rely on disparate datasets, preprocessing pipelines, and evaluation metrics.<n>We present REBench, a comprehensive benchmark dataset for evaluating LLMs on binary reverse engineering tasks.
  arXiv  Detail & Related papers  (2026-04-30T02:03:37Z)
• Agent psychometrics: Task-level performance prediction in agentic coding benchmarks [24.348135523715815]
  We present a framework for predicting success or failure on individual tasks tailored to the agentic coding regime.<n>Our approach augments Item Response Theory (IRT) with rich features extracted from tasks, including issue statements, repository contexts, solutions, and test cases.
  arXiv  Detail & Related papers  (2026-04-01T07:59:59Z)
• CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Cases [40.58765467531474]
  CCR-Bench is a novel benchmark designed to assess large language models' adherence to complex instructions.<n> CCR-Bench is characterized by: (1) deep entanglement of content and formatting requirements in task specifications; (2) instructions that involve intricate task decomposition, conditional reasoning, and procedural planning; and (3) evaluation samples derived entirely from real-world industrial scenarios.
  arXiv  Detail & Related papers  (2026-03-09T01:49:19Z)
• When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference [12.867006554196358]
  We introduce the TSAIA Benchmark, a first attempt to evaluate Large Language Models as time-series AI assistants.<n>The benchmark encompasses a broad spectrum of challenges, ranging from constraint-aware forecasting to anomaly detection with threshold calibration.<n>We apply this benchmark to assess eight state-of-the-art LLMs under a unified evaluation protocol.
  arXiv  Detail & Related papers  (2025-09-01T22:58:57Z)
• EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models [64.70546873396624]
  We present the Extremely Complex Instruction Following Benchmark (EIFBENCH) for evaluating large language models (LLMs)<n>EIFBENCH includes multi-task scenarios that enable comprehensive assessment across diverse task types concurrently.<n>We also propose the Segment Policy Optimization (SegPO) algorithm to enhance the LLM's ability to accurately fulfill multi-task workflow.
  arXiv  Detail & Related papers  (2025-06-10T02:39:55Z)
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
  Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
  arXiv  Detail & Related papers  (2024-06-19T00:28:58Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
  We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation. Specifically, task decomposition, tool selection, and parameter prediction are assessed. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• Can Large Language Models Understand Real-World Complex Instructions? [54.86632921036983]
  Large language models (LLMs) can understand human instructions, but struggle with complex instructions. Existing benchmarks are insufficient to assess LLMs' ability to understand complex instructions. We propose CELLO, a benchmark for evaluating LLMs' ability to follow complex instructions systematically.
  arXiv  Detail & Related papers  (2023-09-17T04:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.