Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce

• URL: http://arxiv.org/abs/2606.12924v1
• Date: Thu, 11 Jun 2026 05:27:42 GMT
• Title: Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce
• Abstract summary: We present a modular two-agent simulation framework for evaluating conversational shopping assistant architectures.<n> Holding the buyer constant across experiments enables controlled comparison of responder designs on identical scenarios.<n>Rolling-window memory outperforms intent-extraction memory on all quality metrics while being 35% faster per query.
• Score: 5.346615622025758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a modular two-agent simulation framework for evaluating conversational shopping assistant architectures. An independent buyer agent, configured with personas, missions, and patience levels, is paired with an interchangeable responder that integrates with a real e-commerce search API. Holding the buyer constant across experiments enables controlled comparison of responder designs on identical scenarios. Using 2011 conversations across 14 persona buckets, we establish four empirical findings. First, rolling-window memory outperforms intent-extraction memory on all quality metrics while being 35% faster per query. Second, illustrating rapid evidence-driven iteration, a systematic failure analysis of a responder version enables targeted fixes that reduce failure and near-failure rates by 62% across the full dataset. Third, swapping the responder LLM backbone from Gemini~2.5 to Llama~3.3~70B costs 0.16--0.45 points despite identical architecture. Finally, we document systematic philosophical disagreement between frontier LLM judges: Gemini rewards process correctness while Claude demands concrete outcomes, despite using the same evaluation prompt.

Related papers

• A Multi-Agent Platform for Automated Enterprise Analytics and Insight Generation [0.0]
  This paper proposes a multi-agent framework built on CrewAI for conversational business intelligence.<n>Five specialized AI agents operate in a sequential pipeline to process natural language queries, retrieve and analyze data, generate visualizations, and deliver actionable insights.
  arXiv  Detail & Related papers  (2026-08-19T09:49:58Z)
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design [0.4499833362998488]
  We introduce a benchmark suite with three evaluation dimensions.<n>We present EngiAI, a Multi-Agent System (MAS) implementation built on LangGraph.
  arXiv  Detail & Related papers  (2026-05-19T12:12:09Z)
• VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation [98.38575149237442]
  VLAA-GUI is a modular GUI framework built around three integrated components.<n>A mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step.<n>A mandatory Loop Breaker provides multi-tier switching interaction mode after repeated failures.
  arXiv  Detail & Related papers  (2026-04-23T07:42:37Z)
• ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search [49.61286310968402]
  We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem.<n>An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous responses.<n>The benchmark comprises 2,691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who), spatial reasoning (Where), and temporal reasoning (When)
  arXiv  Detail & Related papers  (2026-04-14T14:06:19Z)
• APEX-EM: Non-Parametric Online Learning for Autonomous Agents via Structured Procedural-Episodic Experience Replay [7.370176470430802]
  LLM-based autonomous agents lack persistent procedural memory.<n>We present APEX-EM, a non-parametric online learning framework that accumulates, retrieves, and reuses structured procedural plans.
  arXiv  Detail & Related papers  (2026-03-31T00:24:56Z)
• DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles [5.647839536820347]
  We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement to produce well-calibrated confidence estimates.<n>DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline.<n>The learned features generalize across benchmarks with near-zero performance degradation.
  arXiv  Detail & Related papers  (2026-03-21T23:24:12Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Agent4FaceForgery: Multi-Agent LLM Framework for Realistic Face Forgery Detection [108.5042835056188]
  This work introduces Agent4FaceForgery to address two fundamental problems.<n>How to capture the diverse intents and iterative processes of human forgery creation.<n>How to model the complex, often adversarial, text-image interactions that accompany forgeries in social media.
  arXiv  Detail & Related papers  (2025-09-16T01:05:01Z)
• Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving [62.71545696485824]
  We introduce AGENT KB, a universal memory infrastructure enabling seamless experience sharing across heterogeneous agent frameworks without retraining.<n>AGENT KB aggregates trajectories into a structured knowledge base and serves lightweight APIs.<n>We validate AGENT across major frameworks on GAIA, Humanity's Last Exam, GPQA, and SWE-bench.
  arXiv  Detail & Related papers  (2025-07-08T17:59:22Z)
• Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performance [81.05882480184587]
  We propose a computational framework of Kolb's learning cycle with Vygotsky's ZPD for autonomous agents.<n>Agent K is the 1st AI system to successfully integrate Kolb- and Vygotsky-inspired human cognitive learning.<n>With 9 gold, 8 silver, and 12 bronze medals level performance - including 4 gold and 4 silver on prize-awarding competitions - Agent K is the 1st AI system to successfully integrate Kolb- and Vygotsky-inspired human cognitive learning.
  arXiv  Detail & Related papers  (2024-11-05T23:55:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.