PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations

• URL: http://arxiv.org/abs/2605.22855v1
• Date: Tue, 19 May 2026 04:10:05 GMT
• Title: PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations
• Abstract summary: PrefBench is a simulator-based benchmark for hidden-preference personalized pricing negotiations.<n>This paper presents PrefBench, a benchmark for evaluating pricing-agent behavior under hidden buyer preferences.
• Score: 0.18275108630751835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized pricing negotiations are a challenging testbed for LLM agents because successful interaction does not guarantee profitable decision making. A seller may produce valid actions and close many deals while still pricing poorly when buyer willingness to pay and bargaining traits remain hidden. This paper presents PrefBench, a simulator-based benchmark for hidden-preference personalized pricing negotiations. Each episode pairs a simulated buyer with a fixed vehicle-customization bundle; the seller observes public persona descriptors, bundle information, and negotiation history, while latent buyer variables govern valuation, patience, counter-offer behavior, and walkaway decisions. PrefBench evaluates this setting through an LLM-facing state-summary protocol that constrains agents to return strict JSON actions under a fixed hidden-information boundary. We evaluate zero-shot LLM sellers against heuristic references over 7,500 episodes. The tested LLMs follow the protocol reliably and achieve deal rates above 0.99, but their seller-profit outcomes remain weak: the best LLM average profit is only slightly above the random baseline and far below a simple concession heuristic under the same episode stream. These results show that structured action compliance and agreement-seeking behavior can coexist with weak profit-sensitive bargaining. PrefBench provides a controlled benchmark for evaluating pricing-agent behavior under hidden buyer preferences.

Related papers

• When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation [0.39287497907611874]
  We audit this risk in a multi-turn buyer testbed for hotel transactions.<n>Initial implementation reported welfare gains from two marketplace guardrails.<n>A seller-incentive check is non-monotone: increasing profit pressure produces less profit than the default seller prompt.
  arXiv  Detail & Related papers  (2026-09-01T16:48:13Z)
• Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games [50.880420636090896]
  We build an auditable framework that maintains an external belief state over hidden roles in a 9-player Werewolf environment.<n>We report the effect as an association and treat its mechanism as unresolved.
  arXiv  Detail & Related papers  (2026-07-12T16:03:30Z)
• TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate [34.67393151003599]
  Negotiation is a canonical testbed for agentic language models.<n>Unlike math or code, it has no intrinsic verifier.<n>We introduce Terms-Bench, a Bayesian-game framework that makes the environment itself the verifier.
  arXiv  Detail & Related papers  (2026-05-13T06:22:50Z)
• Predicting Decisions of AI Agents from Limited Interaction through Text-Tabular Modeling [20.47884338846419]
  AI agents negotiate and transact in natural language with unfamiliar counterparts.<n>In such interactions, the counterpart's LLM, prompts, control logic, and rule-based fallbacks are hidden.<n>We ask whether an agent can predict an unfamiliar counterpart's next decision from a few interactions.
  arXiv  Detail & Related papers  (2026-05-12T17:09:32Z)
• Training Language Models for Bilateral Trade with Private Information [47.25254982913256]
  Bilateral trade demands individual, strategic surplus rationality, and cooperation to realize gains from trade.<n>We develop a structured bargaining environment where large language models negotiate via tool calls within an event-driven simulator.<n>The environment serves two purposes: a benchmark for frontier models and as a training environment for open-weight models via reinforcement learning.
  arXiv  Detail & Related papers  (2026-04-10T03:04:20Z)
• Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation [33.09732020656996]
  We introduce Conv-FinRe, a benchmark for stock recommendation that evaluates LLMs beyond behavior matching.<n>We build the benchmark from real market data and human decision trajectories, instantiate controlled advisory conversations, and evaluate a suite of state-of-the-art LLMs.<n>Results reveal a persistent tension between rational decision quality and behavioral alignment.
  arXiv  Detail & Related papers  (2026-02-19T01:29:50Z)
• SAND: Boosting LLM Agents with Self-Taught Action Deliberation [54.48979740613828]
  Large Language Model (LLM) agents are commonly tuned with supervised finetuning on ReAct-style expert trajectories or preference optimization over pairwise rollouts.<n>We propose Self-taught ActioN Deliberation (SAND) framework, enabling LLM agents to explicitly deliberate over candidate actions before committing to one.<n>SAND achieves an average 20% improvement over initial supervised finetuning and also outperforms state-of-the-art agent tuning approaches.
  arXiv  Detail & Related papers  (2025-07-10T05:38:15Z)
• Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data [46.65903742010956]
  We present the first large-scale quantitative evaluation of state-of-the-art LLMs' ability to accurately simulate human behavior.<n>Using real-world data from 31,865 online shopping sessions containing 230,965 user actions, our evaluation reveals that prompt-based LLMs achieve only 11.86% accuracy in generating human actions.<n>We also showcase that strategies as simple as fine-tuning LLMs on real human click-through data augmented with synthesized reasoning traces can greatly enhance models' performance.
  arXiv  Detail & Related papers  (2025-03-26T17:33:27Z)
• Differentially Private Steering for Large Language Model Alignment [55.30573701583768]
  We present the first study of aligning Large Language Models with private datasets.<n>Our work proposes the Private Steering for LLM Alignment (PSA) algorithm to edit activations with differential privacy guarantees.<n>Our results show that PSA achieves DP guarantees for LLM alignment with minimal loss in performance.
  arXiv  Detail & Related papers  (2025-01-30T17:58:36Z)
• Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions [77.66677127535222]
  Auto-Arena is an innovative framework that automates the entire evaluation process using LLM-powered agents. In our experiments, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks.
  arXiv  Detail & Related papers  (2024-05-30T17:19:19Z)
• Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method [17.388837360641276]
  This paper describes the Bargaining task as an asymmetric incomplete information game. It allows us to quantitatively assess an agent's performance in the Bargain task. We propose a novel approach called OG-Narrator that integrates a deterministic Offer Generator to control the price range of Buyer's offers.
  arXiv  Detail & Related papers  (2024-02-24T13:36:58Z)
• How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis [50.15061156253347]
  Negotiation is the basis of social interactions; humans negotiate everything from the price of cars to how to share common resources. With rapidly growing interest in using large language models (LLMs) to act as agents on behalf of human users, such LLM agents would also need to be able to negotiate. We develop NegotiationArena: a flexible framework for evaluating and probing the negotiation abilities of LLM agents.
  arXiv  Detail & Related papers  (2024-02-08T17:51:48Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.