Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards

• URL: http://arxiv.org/abs/2604.09855v1
• Date: Fri, 10 Apr 2026 19:35:39 GMT
• Title: Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards
• Abstract summary: We investigate if Reinforcement Learning from Verifiable Rewards can effectively teach Large Language Models to negotiate.<n>We introduce a framework that trains a mid-sized buyer agent against a regulated seller across a wide distribution of real-world products.<n>Our results demonstrate that this verifiable training allows a 30B agent to significantly outperform frontier models over ten times its size in extracting surplus.
• Score: 45.56436052535799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent advancement of Large Language Models (LLMs) has established their potential as autonomous interactive agents. However, they often struggle in strategic games of incomplete information, such as bilateral price negotiation. In this paper, we investigate if Reinforcement Learning from Verifiable Rewards (RLVR) can effectively teach LLMs to negotiate. Specifically, we explore the strategic behaviors that emerge during the learning process. We introduce a framework that trains a mid-sized buyer agent against a regulated LLM seller across a wide distribution of real-world products. By grounding reward signals directly in the maximization of economic surplus and strict adherence to private budget constraints, we reveal a novel four-phase strategic evolution. The agent progresses from naive bargaining to using aggressive starting prices, moves through a phase of deadlock, and ultimately develops sophisticated persuasive skills. Our results demonstrate that this verifiable training allows a 30B agent to significantly outperform frontier models over ten times its size in extracting surplus. Furthermore, the trained agent generalizes robustly to stronger counterparties unseen during training and remains effective even when facing hostile, adversarial seller personas.

Related papers

• Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations [22.41117839056233]
  Negotiation is a fundamental strategic interaction in management science.<n>This paper proposes a specialized training recipe using Reinforcement Learning from Verifiable Rewards.
  arXiv  Detail & Related papers  (2026-07-07T05:41:54Z)
• Counterparty Modeling is Not Strategy: The Limits of LLM Negotiators [29.84830992033132]
  We study whether large language model (LLM) agents do this in a controlled multi-attribute bargaining environment.<n>We find that current LLM agents can model a counterparty's preferences, but do not reliably turn that knowledge into strategic bargaining.<n>Because agents fail to leverage this underlying utility structure for strategic advantage, their final agreements are heavily dictated by surface-level opening anchors rather than actual utility weights.
  arXiv  Detail & Related papers  (2026-05-15T19:27:05Z)
• AIPO: Learning to Reason from Active Interaction [54.10819421625103]
  AIPO enables the policy model to proactively consult three functional collaborative agents when encountering reasoning bottlenecks.<n>AIPO consistently improves reasoning performance, generalizes robustly across different policy models and RLVR algorithms, and effectively expands the reasoning capability boundary of the policy model.
  arXiv  Detail & Related papers  (2026-05-08T19:06:55Z)
• Training Language Models for Bilateral Trade with Private Information [47.25254982913256]
  Bilateral trade demands individual, strategic surplus rationality, and cooperation to realize gains from trade.<n>We develop a structured bargaining environment where large language models negotiate via tool calls within an event-driven simulator.<n>The environment serves two purposes: a benchmark for frontier models and as a training environment for open-weight models via reinforcement learning.
  arXiv  Detail & Related papers  (2026-04-10T03:04:20Z)
• PieArena: Frontier Language Agents Achieve MBA-Level Negotiation Performance and Reveal Novel Behavioral Differences [13.759960839511807]
  We introduce PieArena, a large-scale negotiation benchmark grounded in multi-agent interactions.<n>We find systematic evidence of human-expert-level performance in which a representative frontier language agent (GPT-5) matches or outperforms trained business-school students.
  arXiv  Detail & Related papers  (2026-02-05T04:52:20Z)
• When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training [26.66184262287797]
  Large Language Models (LLMs) often explore suboptimally in sequential decision-making.<n>Recent work has sought to enhance this capability via supervised fine-tuning (SFT) or reinforcement learning (RL), improving regret on the classic multi-armed bandit task.<n>We investigate both paradigms by training LLMs with SFT on expert trajectories and RL with a range of tailored reward signals.<n>The resulting agents outperform pre-trained models and achieve performance comparable to Upper Confidence Bound (UCB) and Thompson Sampling.
  arXiv  Detail & Related papers  (2025-09-29T15:25:42Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis [50.15061156253347]
  Negotiation is the basis of social interactions; humans negotiate everything from the price of cars to how to share common resources. With rapidly growing interest in using large language models (LLMs) to act as agents on behalf of human users, such LLM agents would also need to be able to negotiate. We develop NegotiationArena: a flexible framework for evaluating and probing the negotiation abilities of LLM agents.
  arXiv  Detail & Related papers  (2024-02-08T17:51:48Z)
• MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning [62.065503126104126]
  We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen learning agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. We introduce MERMAIDE, a model-based meta-learning framework to train a principal that can quickly adapt to out-of-distribution agents.
  arXiv  Detail & Related papers  (2023-04-10T15:44:50Z)
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary [86.0846119254031]
  We study the robustness of reinforcement learning with adversarially perturbed state observations. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
  arXiv  Detail & Related papers  (2021-01-21T05:38:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.