Steering Language Models with Game-Theoretic Solvers

• URL: http://arxiv.org/abs/2402.01704v3
• Date: Mon, 16 Dec 2024 11:03:31 GMT
• Title: Steering Language Models with Game-Theoretic Solvers
• Authors: Ian Gemp, Roma Patel, Yoram Bachrach, Marc Lanctot, Vibhavari Dasagi, Luke Marris, Georgios Piliouras, Siqi Liu, Karl Tuyls,
• Abstract summary: We introduce a framework that allows equilibrium solvers to work over the space of natural language dialogue generated by large language models (LLMs) Specifically, by modelling the players, strategies and payoffs in a "game" of dialogue, we create a binding from natural language interactions to the conventional symbolic logic of game theory. We focus on three domains that require different negotiation strategies: scheduling meetings, trading fruit and debate, and evaluate an LLM's generated language when guided by solvers.
• Score: 43.023261136434876
• License:
• Abstract: Mathematical models of interactions among rational agents have long been studied in game theory. However these interactions are often over a small set of discrete game actions which is very different from how humans communicate in natural language. To bridge this gap, we introduce a framework that allows equilibrium solvers to work over the space of natural language dialogue generated by large language models (LLMs). Specifically, by modelling the players, strategies and payoffs in a "game" of dialogue, we create a binding from natural language interactions to the conventional symbolic logic of game theory. Given this binding, we can ask existing game-theoretic algorithms to provide us with strategic solutions (e.g., what string an LLM should generate to maximize payoff in the face of strategic partners or opponents), giving us predictors of stable, rational conversational strategies. We focus on three domains that require different negotiation strategies: scheduling meetings, trading fruit and debate, and evaluate an LLM's generated language when guided by solvers. We see that LLMs that follow game-theory solvers result in dialogue generations that are less exploitable than the control (no guidance from solvers), and the language generated results in higher rewards, in all negotiation domains. We discuss future implications of this work, and how game-theoretic solvers that can leverage the expressivity of natural language can open up a new avenue of guiding language research.

Related papers

• Verbalized Bayesian Persuasion [54.55974023595722]
  Information design (ID) explores how a sender influence the optimal behavior of receivers to achieve specific objectives. This work proposes a verbalized framework in Bayesian persuasion (BP), which extends classic BP to real-world games involving human dialogues for the first time. Numerical experiments in dialogue scenarios, such as recommendation letters, courtroom interactions, and law enforcement, validate that our framework can both reproduce theoretical results in classic BP and discover effective persuasion strategies.
  arXiv  Detail & Related papers  (2025-02-03T18:20:10Z)
• Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game [32.791648070823776]
  Werewolf is a social deduction game that tests language understanding. We develop the Multi-agent Kahneman & Tversky's Optimization (MaKTO) MaKTO achieves a 61% average win rate across various models.
  arXiv  Detail & Related papers  (2025-01-24T04:09:03Z)
• Mastering Board Games by External and Internal Planning with Language Models [30.782334791241556]
  We show that search-based planning can significantly improve LLMs' playing strength across several board games. In external search, the model guides Monte Carlo Tree Search rollouts and evaluations without calls to an external engine, and in internal search, the model directly generates in-context a linearized tree of potential futures. Both build on a language model pre-trained on relevant domain knowledge, capturing the transition and value functions across these games.
  arXiv  Detail & Related papers  (2024-12-02T18:56:51Z)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models [56.25156596019168]
  This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs) Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
  arXiv  Detail & Related papers  (2023-11-30T03:59:31Z)
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations [70.7884839812069]
  Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue.
  arXiv  Detail & Related papers  (2023-11-09T18:45:16Z)
• Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf [19.39740531672788]
  We propose a tuning-free framework to engage large language models in communication games. An empirical study on the representative and widely-studied communication game, Werewolf'', demonstrates that our framework can effectively play Werewolf game without tuning the parameters of the LLMs.
  arXiv  Detail & Related papers  (2023-09-09T01:56:40Z)
• Inner Monologue: Embodied Reasoning through Planning with Language Models [81.07216635735571]
  Large Language Models (LLMs) can be applied to domains beyond natural language processing. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios.
  arXiv  Detail & Related papers  (2022-07-12T15:20:48Z)
• Emergent Communication of Generalizations [13.14792537601313]
  We argue that communicating about a single object in a shared visual context is prone to overfitting and does not encourage language useful beyond concrete reference. We propose games that require communicating generalizations over sets of objects representing abstract visual concepts. We find that these games greatly improve systematicity and interpretability of the learned languages.
  arXiv  Detail & Related papers  (2021-06-04T19:02:18Z)
• I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents [69.68400056148336]
  We train a goal-oriented model with reinforcement learning against an imitation-learned chit-chat'' model. We show that both models outperform an inverse model baseline and can converse naturally with their dialogue partner in order to achieve goals.
  arXiv  Detail & Related papers  (2020-02-07T16:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.