Training Language Models for Bilateral Trade with Private Information

• URL: http://arxiv.org/abs/2604.16472v1
• Date: Fri, 10 Apr 2026 03:04:20 GMT
• Title: Training Language Models for Bilateral Trade with Private Information
• Abstract summary: Bilateral trade demands individual, strategic surplus rationality, and cooperation to realize gains from trade.<n>We develop a structured bargaining environment where large language models negotiate via tool calls within an event-driven simulator.<n>The environment serves two purposes: a benchmark for frontier models and as a training environment for open-weight models via reinforcement learning.
• Score: 47.25254982913256
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bilateral bargaining under incomplete information provides a controlled testbed for evaluating large language model (LLM) agent capabilities. Bilateral trade demands individual rationality, strategic surplus maximization, and cooperation to realize gains from trade. We develop a structured bargaining environment where LLMs negotiate via tool calls within an event-driven simulator, separating binding offers from natural-language messages to enable automated evaluation. The environment serves two purposes: as a benchmark for frontier models and as a training environment for open-weight models via reinforcement learning. In benchmark experiments, a round-robin tournament among five frontier models (15,000 negotiations) reveals that effective strategies implement price discrimination through sequential offers. Aggressive anchoring, calibrated concession, and temporal patience correlate with the highest surplus share and deal rate. Accommodating strategies that concede quickly disable price discrimination in the buyer role, yielding the lowest surplus capture and deal completion. Stronger models scale their behavior proportionally to item value, maintaining performance across price tiers; weaker models perform well only when wide zones of possible agreement offset suboptimal strategies. In training experiments, we fine-tune Qwen3 (8B, 14B) via supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) against a fixed frontier opponent. These stages optimize competing objectives: SFT approximately doubles surplus share but reduces deal rates, while RL recovers deal rates but erodes surplus gains, reflecting the reward structure. SFT also compresses surplus variation across price tiers, which generalizes to unseen opponents, suggesting that behavioral cloning instills proportional strategies rather than memorized price points.

Related papers

• World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning [60.88859946006306]
  Adversa World Modeling is a theoretically grounded multi-agent self-play fine-tuning framework.<n>In inner minimization, the planner's predictive world model is converted into a role-conditioned adversary that learns sparse, scene-adaptive attack coalitions.<n> Experiments on the nuPlan and InterPlan benchmarks demonstrate that our method generates transferable adversarial interactions.
  arXiv  Detail & Related papers  (2026-07-12T07:44:16Z)
• Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations [22.41117839056233]
  Negotiation is a fundamental strategic interaction in management science.<n>This paper proposes a specialized training recipe using Reinforcement Learning from Verifiable Rewards.
  arXiv  Detail & Related papers  (2026-07-07T05:41:54Z)
• A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctions [19.323612677924732]
  We introduce the A3M framework, which integrates adaptive deep reinforcement learning, explicit adversarial reasoning, and principled multi-objective reward design for online auction strategy optimization.<n>A3M employs an actor-critic DRL backbone to dynamically balance exploration and exploitation, an opponent model for fictitious play against non-stationary adversaries, and a composite reward function to jointly maximize utility, auctioneer revenue, and fairness.<n>Results show that A3M reduces final regret by 30--40% in standard settings, maintains robust performance against adversarial strategy shifts, scales favorably with the number of units $K$, and enables tunable multi-objective trade
  arXiv  Detail & Related papers  (2026-06-27T14:30:43Z)
• GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization [54.596224644751565]
  Post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities.<n>To address this, existing methods such as Group reward-Decoupled Policy Optimization (GDPO) decompose the overall score into independent reward groups, then compute the RL loss separately within each group.<n>We propose Group-Dynamic reward-Decoupled Policy Optimization (GD$2$PO), which employs a conflict-aware filtering mechanism to mask out rollouts suffering from severe reward-wise disagreement.
  arXiv  Detail & Related papers  (2026-06-15T14:19:28Z)
• Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining [0.5691230599672109]
  We introduce textscCattle Trade, a benchmark for evaluating large language models (LLMs) as agents in strategic reasoning.<n>The benchmark combines auctions, hidden-offer trade challenges (TCs), bargaining, bluffing, opponent modeling, and resource allocation within a single long-horizon game.<n>We evaluate seven cost-efficient language models and three deterministic code agents across 242 games.
  arXiv  Detail & Related papers  (2026-05-14T08:20:03Z)
• Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards [45.56436052535799]
  We investigate if Reinforcement Learning from Verifiable Rewards can effectively teach Large Language Models to negotiate.<n>We introduce a framework that trains a mid-sized buyer agent against a regulated seller across a wide distribution of real-world products.<n>Our results demonstrate that this verifiable training allows a 30B agent to significantly outperform frontier models over ten times its size in extracting surplus.
  arXiv  Detail & Related papers  (2026-04-10T19:35:39Z)
• Decomposable Reward Modeling and Realistic Environment Design for Reinforcement Learning-Based Forex Trading [0.0]
  This paper presents a modular reinforcement learning framework for Forex trading.<n>It includes a friction-aware execution engine that enforces strict anti-lookahead semantics, with observations at time t, execution at time t+1, and mark-to-market at time t+1.<n>It also incorporates realistic costs such as spread, commission, slippage, rollover financing, and margin-triggered liquidation.
  arXiv  Detail & Related papers  (2026-03-20T13:38:19Z)
• Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity [45.92643973404507]
  We investigate whether aligned models are vulnerable to Preference-Undermining Attacks (PUA), a class of manipulative prompting strategies.<n>Surprisingly, more advanced models are sometimes more susceptible to manipulative prompts.
  arXiv  Detail & Related papers  (2026-01-10T15:16:23Z)
• Tacit Bidder-Side Collusion: Artificial Intelligence in Dynamic Auctions [0.0]
  We study whether large language models acting as autonomous bidders can tacitly collude by coordinating when to accept platform posted payouts in repeated Dutch auctions.<n>We present a minimal repeated auction model that yields a simple incentive compatibility condition and a closed form threshold for sustainable collusion for Nash equilibria.
  arXiv  Detail & Related papers  (2025-11-26T18:32:18Z)
• Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets [57.179679246370114]
  In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices.<n>During deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact.<n>Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties.<n>We develop a novel class of elliptic uncertainty sets, enabling efficient and tractable robust policy evaluation.
  arXiv  Detail & Related papers  (2025-10-22T18:22:25Z)
• The Bidding Games: Reinforcement Learning for MEV Extraction on Polygon Blockchain [0.11880231424287215]
  We present a reinforcement learning framework for MEV extraction on Polygon Atlas.<n>Our work establishes that reinforcement learning provides a critical advantage in high-frequency MEV environments.
  arXiv  Detail & Related papers  (2025-10-16T12:54:53Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off [107.35833747750446]
  adversarial examples can be crafted by adding imperceptible perturbations to legitimate documents. This vulnerability raises significant concerns about their reliability and hinders the widespread deployment of NRMs. In this study, we establish theoretical guarantees regarding the effectiveness-robustness trade-off in NRMs.
  arXiv  Detail & Related papers  (2023-12-16T05:38:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.