Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State

• URL: http://arxiv.org/abs/2605.06529v1
• Date: Thu, 07 May 2026 16:31:39 GMT
• Title: Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State
• Abstract summary: We study failure in a two-hotel revenue-management simulator.<n>Hotel A trains an agent against a fixed rule-based revenue-management competitor.
• Score: 0.39287497907611874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Outcome metrics can certify the wrong behavior. We study this failure in a two-hotel revenue-management simulator where Hotel A trains an agent against a fixed rule-based revenue-management competitor, Hotel B. A standard learning agent can obtain near-reference revenue per available room (RevPAR) while failing to learn market-like yield management: it sells too aggressively, undercuts, or collapses to modal price buckets. We diagnose this as a Goodhart-style failure under partial observability. Hotel A cannot observe the competitor's remaining inventory, booking curve, or pricing rule, so the same Hotel A-visible state maps to multiple plausible Hotel B prices. Deterministic value-based RL and deterministic copying collapse this unresolved uncertainty into shortcut behavior. We introduce a trace-level diagnostic protocol using RevPAR, occupancy, ADR, full price-bucket distributions, L1/JS distances, and seed-level confidence intervals. The verified repair is Trace-Prior RL: learn a distributional market prior from lagged market traces, then train a stochastic pricing policy with a RevPAR reward and a KL penalty to the learned prior. The final policy matches Hotel B's RevPAR, occupancy, ADR, and price distribution within seed-level uncertainty, while still optimizing Hotel A's own reward. We argue that the contribution is not a new optimizer and not a hotel-pricing leaderboard, but a reproducible failure-and-repair recipe for agentic systems where scalar rewards are easy to game and the intended behavior is only visible in traces. A key finding is that higher exact action accuracy can worsen aggregate trace alignment when the target is distributional.

Related papers

• When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation [0.39287497907611874]
  We audit this risk in a multi-turn buyer testbed for hotel transactions.<n>Initial implementation reported welfare gains from two marketplace guardrails.<n>A seller-incentive check is non-monotone: increasing profit pressure produces less profit than the default seller prompt.
  arXiv  Detail & Related papers  (2026-09-01T16:48:13Z)
• When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits [11.731565558821835]
  Personalizing marketing messages with contextual multi-armed bandits (CMABs) drives real business value.<n>However, the objective that ultimately matters is observed only weeks later, too late to drive online learning.<n>We give an ordered diagnostic protocol that screens a reward-and-policy candidate on two axes, alignment and learnability.
  arXiv  Detail & Related papers  (2026-08-12T01:53:21Z)
• Detecting an Effect Is Not Learning to Act on It: A Reward-SNR Floor for LLM Acquisition Agents [1.5116447879757695]
  Pipelines can pay a per-example cost to acquire an auxiliary, model-derived observation, and then must decide when the acquired signal is worth using.<n>Our thesis is a distinction that is easy to miss: detecting that such a signal helps on average is not the same as learning to act on it per instance.<n>We explain this with one distinction, detecting a mean effect vs. learning a per-instance acquisition policy, and a reward-SNR detectability floor.
  arXiv  Detail & Related papers  (2026-08-11T03:46:23Z)
• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
  Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
  arXiv  Detail & Related papers  (2026-07-29T00:19:09Z)
• When Outcome Looks Right But Discipline Fails: Trace-Based Evaluation Under Hidden Competitor State [0.39287497907611874]
  Outcome-only evaluation can certify economically unsafe agents.<n>In hotel pricing with hidden competitor state, a learner can achieve plausible revenue per available room while failing to preserve the rate discipline of a rule-based revenue-management policy.<n>We introduce discipline stability, a trace-based evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T15:58:34Z)
• AI Alignment via Incentives and Correction [14.986111703734222]
  We study AI alignment through the lens of law-and-economics models of deterrence and enforcement.<n>A solver may benefit from producing a persuasive but incorrect answer, hiding uncertainty, or exploiting spurious shortcuts.<n>We formalize this interaction in a two-agent model in which a principal chooses rewards over joint correction outcomes.
  arXiv  Detail & Related papers  (2026-05-02T23:28:02Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models [68.45272703833209]
  We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
  arXiv  Detail & Related papers  (2026-02-20T23:38:03Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• LLM Collusion [5.363252654303049]
  Large language models (LLMs) can facilitate collusion in a duopoly when both sellers rely on the same pre-trained model.<n>We show that configuring LLMs for robustness and robustness can induce collusion via a phase transition.
  arXiv  Detail & Related papers  (2026-01-03T20:38:21Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation [65.4532392602682]
  One of the main challenges in offline Reinforcement Learning (RL) is the distribution shift that arises from the learned policy deviating from the data collection policy. This is often addressed by avoiding out-of-distribution (OOD) actions during policy improvement as their presence can lead to substantial performance degradation. We introduce AlberDICE, an offline MARL algorithm that performs centralized training of individual agents based on stationary distribution optimization.
  arXiv  Detail & Related papers  (2023-11-03T18:56:48Z)
• Decoupled Kullback-Leibler Divergence Loss [90.54331083430597]
  We prove that the Kullback-Leibler (KL) Divergence loss is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss. We introduce class-wise global information into KL/DKL to bias from individual samples. The proposed approach achieves new state-of-the-art adversarial robustness on the public leaderboard.
  arXiv  Detail & Related papers  (2023-05-23T11:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.