When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation

• URL: http://arxiv.org/abs/2609.01519v1
• Date: Tue, 01 Sep 2026 16:48:13 GMT
• Title: When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation
• Abstract summary: We audit this risk in a multi-turn buyer testbed for hotel transactions.<n>Initial implementation reported welfare gains from two marketplace guardrails.<n>A seller-incentive check is non-monotone: increasing profit pressure produces less profit than the default seller prompt.
• Score: 0.39287497907611874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive simulations increasingly evaluate policies in markets populated by language-model agents. Their outputs can look economic---prices, profits, consumer surplus, and welfare---without instantiating the behavior named in the claim. We audit this risk in a multi-turn buyer--seller testbed for configurable hotel transactions. An initial implementation reported welfare gains from two marketplace guardrails of +87.4, +35.0, and +28.8 across a Qwen2.5 1.5B--14B ladder. It also gave guarded and unguarded agents different offer schemas and choice procedures. Holding the schema and buyer chooser fixed changes the paired contrasts to +7.2, -13.9, and +23.8. The four largest 14B single-generation effects averaged +229; after three generations per profile-condition, they averaged +37.6 (95% bootstrap interval [-34.2, 109.3]), while generation residuals account for 49.9% of variation in this post-hoc probe. A seller-incentive check is non-monotone: increasing profit pressure produces less profit than the default seller prompt. Scripted positive controls show why this matters. A profit-maximizing seller already attains first-best welfare, so guardrails mostly redistribute and reduce welfare; they create welfare only when the seller is explicitly programmed to force inefficient bundles. We contribute a construct-validity contract separating incentive validity, protocol isolation, stochastic stability, and welfare accounting, and returning INVALID or INCONCLUSIVE before substantive policy claims. In our case, the original estimate is INVALID under protocol isolation, while the controlled study remains INCONCLUSIVE under incentive validity and stochastic stability. The case does not show that guardrails are ineffective; it shows their apparent value is unidentified until the simulated agents and protocol pass these checks.

Related papers

• BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure [47.61632934139226]
  LM-agent benchmarks increasingly function as interactive evaluation infrastructure.<n>This interactivity makes evaluations vulnerable to reward hacking.<n>This paper presents BenchShield, a model-backed instrumentation layer for reward integrity.
  arXiv  Detail & Related papers  (2026-09-10T03:10:58Z)
• VST: Verifiable Structured Transport for Auditable Agent-to-Agent Alpha Discovery [11.790070440436294]
  Agent-to-agent (A2A) alpha discovery is slowed by repeated feedback cycles between mining and evaluation agents.<n>We first restructure this communication as a structured agent-to-agent protocol of emphtyped, causally addressable, unicast records<n>What typing provides is a state that can be schema-checked, replayed deterministically, and prevented by construction from leaking a forecast to an evaluator.
  arXiv  Detail & Related papers  (2026-09-07T05:43:50Z)
• Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence [0.0]
  Two instances of one model, in a two-agent handoff, co-fail on 90.0% of the missions on which either fails.<n>Substituting a different model reduces the association in six of six contrasts.
  arXiv  Detail & Related papers  (2026-08-13T07:25:05Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games [50.880420636090896]
  We build an auditable framework that maintains an external belief state over hidden roles in a 9-player Werewolf environment.<n>We report the effect as an association and treat its mechanism as unresolved.
  arXiv  Detail & Related papers  (2026-07-12T16:03:30Z)
• Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis [2.7092559671391]
  Agentic LLM systems such as OpenScholar and PaperQA2 read the scientific literature and return cited answers.<n>Neither audits the reliability of that check itself.<n>We show it is not reliable, and that this matters.<n>We present a gold-anchored evaluation protocol and a deployable guard.
  arXiv  Detail & Related papers  (2026-07-10T02:05:17Z)
• PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations [0.18275108630751835]
  PrefBench is a simulator-based benchmark for hidden-preference personalized pricing negotiations.<n>This paper presents PrefBench, a benchmark for evaluating pricing-agent behavior under hidden buyer preferences.
  arXiv  Detail & Related papers  (2026-05-19T04:10:05Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Neither Consent nor Property: A Policy Lab for Data Law [4.588656982178105]
  This paper makes the opaque data market in the AI economy empirically legible for the first time.<n>The pipeline begins with multi-year fieldwork to extract the market's hidden logic.<n>It is validated against reality, reproducing observed trade patterns.
  arXiv  Detail & Related papers  (2025-10-30T17:27:03Z)
• Beyond Linear Probes: Dynamic Safety Monitoring for Language Models [67.15793594651609]
  Traditional safety monitors require the same amount of compute for every query.<n>We introduce Truncated Polynomials (TPCs), a natural extension of linear probes for dynamic activation monitoring.<n>Our key insight is that TPCs can be trained and evaluated progressively, term-by-term.
  arXiv  Detail & Related papers  (2025-09-30T13:32:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.