Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models

• URL: http://arxiv.org/abs/2603.10098v1
• Date: Tue, 10 Mar 2026 17:37:06 GMT
• Title: Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models
• Abstract summary: Deep reinforcement learning oracles produce black-box' neural network policies that are difficult to interpret, trust or debug.<n>We introduce Code-Space Response Oracles (CSRO), a novel framework that addresses this challenge by replacing RL oracles with Large Language Models (LLMs)<n>Our work presents a new perspective on multi-agent learning, shifting the focus from optimizing opaque policy parameters to interpretable algorithmic behavior.
• Score: 8.649235365712004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in multi-agent reinforcement learning, particularly Policy-Space Response Oracles (PSRO), have enabled the computation of approximate game-theoretic equilibria in increasingly complex domains. However, these methods rely on deep reinforcement learning oracles that produce `black-box' neural network policies, making them difficult to interpret, trust or debug. We introduce Code-Space Response Oracles (CSRO), a novel framework that addresses this challenge by replacing RL oracles with Large Language Models (LLMs). CSRO reframes the best response computation as a code generation task, prompting an LLM to generate policies directly as human-readable code. This approach not only yields inherently interpretable policies but also leverages the LLM's pretrained knowledge to discover complex, human-like strategies. We explore multiple ways to construct and enhance an LLM-based oracle: zero-shot prompting, iterative refinement and \emph{AlphaEvolve}, a distributed LLM-based evolutionary system. We demonstrate that CSRO achieves performance competitive with baselines while producing a diverse set of explainable policies. Our work presents a new perspective on multi-agent learning, shifting the focus from optimizing opaque policy parameters to synthesizing interpretable algorithmic behavior.

Related papers

• When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks? [15.079504478885953]
  Large language models (LLMs) can serve as effective black-box policy models for reinforcement learning tasks.<n>We introduce Prompted Policy Optimization (PromptPO), an iterative method that prompts an LLM with Python descriptions of the state space, action space, and reward function.<n>PromptPO often matches or exceeds the performance of standard RL baselines while using substantially fewer environment interactions.
  arXiv  Detail & Related papers  (2026-05-29T01:24:24Z)
• Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs [53.412166189410904]
  We use supervised learning as a computational proxy to establish a hierarchy of commonly adopted RL oracles under low-rank Markov Decision Processes (MDPs)<n>Motivated by this observation, we propose a novel optimistic actor-critic algorithm that relies solely on the policy evaluation oracle.<n>We prove that our algorithm outperforms the existing sample complexity guarantees for low-rank MDPs while avoiding computationally expensive planning or optimization oracles commonly assumed in prior works.
  arXiv  Detail & Related papers  (2026-05-02T04:46:54Z)
• Poly-EPO: Training Exploratory Reasoning Models [62.82992914206963]
  We present a framework for post-training language models (LMs) that explicitly encourages optimistic exploration and promotes a synergy between exploration and exploitation.<n>We propose Polychromic Exploratory Policy Optimization (Poly-EPO), which instantiates this framework with an objective that explicitly synergizes exploration and exploitation.
  arXiv  Detail & Related papers  (2026-04-19T22:54:19Z)
• Policy-Conditioned Policies for Multi-Agent Task Solving [53.67744322553693]
  In this work, we propose a paradigm shift that bridges the gap by representing policies as human-interpretable source code.<n>We reformulate the learning problem by utilizing Large Language Models (LLMs) as approximate interpreters.<n>We formalize this process as textitProgrammatic Iterated Best Response (PIBR), an algorithm where the policy code is optimized by textual gradients.
  arXiv  Detail & Related papers  (2025-12-24T07:42:10Z)
• Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques [14.892995952768352]
  Language Models (LMs) have excelled at tasks like text generation, summarization, and question answering.<n>Their inference remains computationally expensive and energy intensive in settings with limited hardware, power, or bandwidth.<n>Recent approaches have introduced multi LLM intelligent model selection strategies that dynamically allocate computational resources based on query complexity.
  arXiv  Detail & Related papers  (2025-06-06T23:13:08Z)
• Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey [14.942504483776432]
  reinforcement learning (RL) has emerged as a pivotal technique for code generation and optimization.<n>This paper presents a systematic survey of the application of RL in code optimization and generation.
  arXiv  Detail & Related papers  (2024-12-29T06:15:41Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Nash Learning from Human Feedback [86.09617990412941]
  We introduce an alternative pipeline for the fine-tuning of large language models using pairwise human feedback. We term this approach Nash learning from human feedback (NLHF) We present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent.
  arXiv  Detail & Related papers  (2023-12-01T19:26:23Z)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models [56.25156596019168]
  This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs) Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
  arXiv  Detail & Related papers  (2023-11-30T03:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.