PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants

• URL: http://arxiv.org/abs/2608.26180v1
• Date: Thu, 20 Aug 2026 01:21:51 GMT
• Title: PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants
• Abstract summary: Shopping assistants are shifting from ranked product lists toward structured decision support, where systems must synthesize shopper context, product evidence, and next-step guidance into a coherent recommendation experience.<n>Existing personalization, grounding, and LLM-as-a-judge benchmarks cover pieces of this problem, but they do not define a joint evaluation target for structured shopping-assistant responses.<n>We formulate this missing evaluation target as PACE: Personalized, Actionable, Compositional, and Evidence-grounded evaluation.<n>We instantiate PACE with two artifacts: PACEShop, a benchmark dataset that makes the target measurable through 22,625 controlled records with structured personas, audit
• Score: 36.11628216058589
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Shopping assistants are shifting from ranked product lists toward structured decision support, where systems must synthesize shopper context, product evidence, and next-step guidance into a coherent recommendation experience. This changes the unit of evaluation: a fluent response can still fail by ignoring shopper context, contradicting itself across components, or leaving defects too vague to localize. Existing personalization, grounding, and LLM-as-a-judge benchmarks cover pieces of this problem, but they do not define a joint evaluation target for structured shopping-assistant responses. We formulate this missing evaluation target as PACE: Personalized, Actionable, Compositional, and Evidence-grounded evaluation. We instantiate PACE with two artifacts: PACEShop, a benchmark dataset that makes the target measurable through 22,625 controlled records with structured personas, auditable evidence pools, GOOD/BAD labels, and gold defect family and location annotations; and PACEJudge, a training-free judging protocol that makes the target reportable through a structured output contract. Our experiments show that generic judges can recognize broad quality but fail to recover the diagnostic fields required for PACE; PACEShop makes these failures verifiable, and PACEJudge improves persona-source, cross-component, grounding, and family/location closure without retraining, showing that realistic shopping-assistant evaluation requires a task-matched output contract rather than only a stronger backbone or scalar prompt.

Related papers

• Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification [5.646213674986304]
  A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score.<n>We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters.<n>Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search.
  arXiv  Detail & Related papers  (2026-09-14T13:35:32Z)
• ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation [54.82704239643649]
  We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans.<n>Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback.<n>We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness.
  arXiv  Detail & Related papers  (2026-09-08T17:30:16Z)
• SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition [0.30586855806896046]
  We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions.<n>On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist.
  arXiv  Detail & Related papers  (2026-08-18T20:27:56Z)
• Context Is Not Authority: Structured Runtime Governance for Financial Market Agents [13.810351865063632]
  SAGE-Fin is a finance-specific authority-handoff contract.<n>It compiles proposals into typed, adapter-bound candidates.<n>It requires an exact-artifact receipt whose nominal type matches the consuming response, execution, or policy adapter.
  arXiv  Detail & Related papers  (2026-08-10T02:26:07Z)
• UXBench: Measuring the Actionability of LLM-Generated UX Critiques [58.14408744110387]
  We introduce UXBench, a benchmark for evaluating large language models (LLMs) as interaction-grounded UX judges.<n> UXBench comprises local-first runnable web fixtures spanning ten product-surface families, paired with coverage-gated rubric exploration.<n>We evaluate eight frontier models under both an automated repair-lift protocol and a blind human validation study.
  arXiv  Detail & Related papers  (2026-06-15T06:08:39Z)
• Unsupervised Skill Discovery for Agentic Data Analysis [55.42853694128929]
  DataCOPE is an unsupervised verifier-guided skill discovery framework for data-analytic agents.<n>It iteratively coordinates a Data-Analytic Agent for trajectory generation, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive skill distillation.<n>We evaluate DataCOPE on report-style analysis from Deep Data Research and reasoning-style analysis from DABStep.
  arXiv  Detail & Related papers  (2026-06-04T17:20:47Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation [56.84819098277464]
  CoNL is a framework that unifies generation, evaluation, and meta-evaluation through multi-agent self-play.<n>CoNL achieves consistent improvements over self-rewarding baselines while maintaining stable training.
  arXiv  Detail & Related papers  (2026-01-29T09:41:14Z)
• ORCHID: Orchestrated Retrieval-Augmented Classification with Human-in-the-Loop Intelligent Decision-Making for High-Risk Property [6.643427585499247]
  ORCHID is a modular agentic system for HRP classification.<n>It pairs retrieval-augmented generation (RAG) with human oversight to produce policy-based outputs that can be audited.<n>The demonstration shows single item submission, grounded citations, SME feedback capture, and exportable audit artifacts.
  arXiv  Detail & Related papers  (2025-11-07T03:48:05Z)
• FOR-Prompting: From Objection to Revision via an Asymmetric Prompting Protocol [7.765950922513099]
  Reasoning protocols organize internal deliberation but lack an explicit mechanism for external questioning that elicits self-revision.<n>We present FOR-Prompting, an asymmetric protocol where a Defender proposes an answer, an Objectioner raises question-style objections with no direct fixes, and a Host enforces consistency and closure.<n>On GSM8K we observe about a 22% point gain over single-prompt and accuracy on par with CoT, with more than 10% higher ratings in reasoning and coherence from a uniform GPT 4.1 judge.
  arXiv  Detail & Related papers  (2025-10-02T04:57:58Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.