xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

• URL: http://arxiv.org/abs/2609.07784v1
• Date: Mon, 07 Sep 2026 17:30:58 GMT
• Title: xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems
• Abstract summary: We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities.<n>The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements.<n>The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points.
• Score: 57.62172039072062
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world requests are often open-ended, casually specified, and context-dependent, requiring models not only to follow explicit instructions but also to infer unstated needs from user background and situational context. We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities. The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements. We evaluate 11 frontier models under standardized agentic settings. The best models achieve a task-level score of 75.6\%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points. These results reveal implicit requirement inference as a persistent bottleneck for reliably satisfying real-world everyday user needs.

Related papers

• AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use [38.67550182213879]
  AppWorld-UL is a benchmark of 516 challenging tasks requiring diverse agent-user interactions.<n>Our evaluation reveals that a state-of-the-art LLM, Claude Opus 4.7, achieves only 48.6% success on AppWorld-UL.<n>This demonstrates the benchmark's difficulty and its potential to advance research on user-in-the-loop tool-use agents.
  arXiv  Detail & Related papers  (2026-07-10T16:55:04Z)
• Automatically Evolving Prompt Guidelines for Task-Specific Optimization [12.329476612271442]
  In large language models, user queries are often underspecified.<n>Existing prompt engineering guidelines aim to mitigate this issue, they are typically generic and task-agnostic.<n>We propose AGOPS, an automatic approach that evolves task-specific guidelines via an optimization scheme.
  arXiv  Detail & Related papers  (2026-05-07T15:50:42Z)
• CCTU: A Benchmark for Tool Use under Complex Constraints [66.87622847854337]
  We introduce CCTU, a benchmark for evaluating large language models (LLMs) under complex constraints.<n>The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios.<n>We develop an executable constraint validation module that performs step-level validation and enforces compliance.
  arXiv  Detail & Related papers  (2026-03-16T14:05:13Z)
• LiveAgentBench: Comprehensive Benchmarking of Agentic Systems Across 104 Real-World Challenges [34.17635007594549]
  We present LiveAgentBench, a comprehensive benchmark with 104 scenarios that reflect real user requirements.<n>It is constructed from publicly sourced questions on social media and real-world products.<n>This release includes 374 tasks, with 125 for validation and 249 for testing.
  arXiv  Detail & Related papers  (2026-03-03T04:03:05Z)
• GISA: A Benchmark for General Information-Seeking Assistant [102.30831921333755]
  GISA is a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries.<n>It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization.<n>Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score.
  arXiv  Detail & Related papers  (2026-02-09T11:44:15Z)
• CL-bench: A Benchmark for Context Learning [152.2879060355882]
  We term this capability context learning, a crucial ability that humans naturally possess but has been largely overlooked.<n>We introduce CL-bench, a real-world benchmark consisting of 500 contexts, 1,899 tasks, and 31,607 verifications.<n> CL-bench represents a step towards building LMs with this fundamental capability, making them more intelligent and advancing their deployment in real-world scenarios.
  arXiv  Detail & Related papers  (2026-02-03T14:37:47Z)
• What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models [10.883552856100684]
  We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities.<n>We find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries.
  arXiv  Detail & Related papers  (2026-01-07T02:33:03Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests [0.6249768559720121]
  We develop and release a novel Arithmetic Constraint-Satisfaction (ACS) benchmarking dataset. This dataset consists of complex user requests with corresponding constraints, agent responses and human labels indicating each constraint's satisfaction level in the response. We show that most models still have a significant headroom for improvement, and that errors primarily stem from reasoning issues.
  arXiv  Detail & Related papers  (2024-09-22T09:27:42Z)
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
  Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
  arXiv  Detail & Related papers  (2024-06-19T00:28:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.