EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

• URL: http://arxiv.org/abs/2606.17698v2
• Date: Tue, 23 Jun 2026 08:51:01 GMT
• Title: EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
• Abstract summary: EComAgentBench is a benchmark of 662 tasks grounded in real Amazon products and reviews.<n>Each task scatters requirements across a visible query, a tool-gated profile, and scripted clarification.<n> typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source.
• Score: 8.1223119066448
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.

Related papers

• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• Agentic Commerce World: An Auditable and Verifiable Environment for Vibe Commerce [83.90449137361595]
  Vibe commerce allows people to express buying or selling goals in natural language and delegate tasks to AI agents.<n>We introduce Agentic Commerce World (ACWorld), an environment for evaluating such agents across ongoing transactions.<n>ACWorld validates agent actions before updating shared transaction state and records the resulting interactions, making agent behavior auditable and evaluation reproducible.
  arXiv  Detail & Related papers  (2026-08-03T16:20:17Z)
• SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration [77.28003194268977]
  SearchOS is a system-level multi-agent framework that turns fragile, implicit search progress into explicit, persistent, and shared state.<n>Built on SOCM, SearchOS applies a pipeline-parallel scheduling mechanism that overlaps the execution of sub-agents.<n>SearchOS leads all metrics among the evaluated single- and multi-agent baselines.
  arXiv  Detail & Related papers  (2026-07-16T17:51:23Z)
• Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents [2.3488056916440856]
  Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified.<n>We introduce Ambig-DS, two diagnostic suites: one for prediction-target ambiguity and one for evaluation-objective ambiguity.<n>For every task we pair the original, fully specified version with an ambiguous variant produced by controlled edits.
  arXiv  Detail & Related papers  (2026-05-10T18:34:12Z)
• CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend [2.9612444540570113]
  We present CUJBench, the first benchmark to combine browser-visible failure evidence with backend observability in a diagnostic framing.<n>The benchmark yields an overall accuracy of 19.7% with a ceiling of 52%, well below saturation.
  arXiv  Detail & Related papers  (2026-04-25T22:10:53Z)
• AgentSearchBench: A Benchmark for AI Agent Search in the Wild [21.739444037726418]
  We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild.<n>The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions.<n>We show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality.
  arXiv  Detail & Related papers  (2026-04-24T10:53:54Z)
• Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents [4.301199871195023]
  Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering.<n>We evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified.<n>We propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution.
  arXiv  Detail & Related papers  (2026-03-27T09:56:26Z)
• GISA: A Benchmark for General Information-Seeking Assistant [102.30831921333755]
  GISA is a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries.<n>It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization.<n>Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score.
  arXiv  Detail & Related papers  (2026-02-09T11:44:15Z)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
  Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
  arXiv  Detail & Related papers  (2026-01-17T01:29:30Z)
• SmartSnap: Proactive Evidence Seeking for Self-Verifying Agents [45.71333459905404]
  SmartSnap is a paradigm shift from passive, post-hoc verification to proactive, in-situ self-verification by the agent itself.<n>We introduce the Self-Verifying Agent, a new type of agent designed with dual missions: to complete a task and to prove its accomplishment with curated evidences.<n>Experiments on mobile tasks across model families and scales demonstrate that our SmartSnap paradigm allows training LLM-driven agents in a scalable manner.
  arXiv  Detail & Related papers  (2025-12-26T14:51:39Z)
• ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions [68.81939215223818]
  ProductAgent is a conversational information seeking agent equipped with abilities of strategic clarification question generation and dynamic product retrieval. We develop the agent with strategies for product feature summarization, query generation, and product retrieval. Experiments show that ProductAgent interacts positively with the user and enhances retrieval performance with increasing dialogue turns.
  arXiv  Detail & Related papers  (2024-07-01T03:50:23Z)
• InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks [84.7788065721689]
  In this paper, we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks. This benchmark contains DAEval, a dataset consisting of 257 data analysis questions derived from 52 CSV files. Building on top of our agent framework, we develop a specialized agent, DAAgent, which surpasses GPT-3.5 by 3.9% on DABench.
  arXiv  Detail & Related papers  (2024-01-10T19:04:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.