Multi-Agent Reinforcement Learning for Market Making: Competition without Collusion

• URL: http://arxiv.org/abs/2510.25929v1
• Date: Wed, 29 Oct 2025 20:07:47 GMT
• Title: Multi-Agent Reinforcement Learning for Market Making: Competition without Collusion
• Authors: Ziyi Wang, Carmine Ventre, Maria Polukarov,
• Abstract summary: We propose a hierarchical multi-agent reinforcement learning framework to study algorithmic collusion in market making.<n>The framework includes a self-interested market maker (AgentA), which is trained in an uncertain environment shaped by an adversary.<n>We show that adaptive incentive control supports more sustainable strategic co-existence in heterogeneous agent environments.
• Score: 6.598173855286935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Algorithmic collusion has emerged as a central question in AI: Will the interaction between different AI agents deployed in markets lead to collusion? More generally, understanding how emergent behavior, be it a cartel or market dominance from more advanced bots, affects the market overall is an important research question. We propose a hierarchical multi-agent reinforcement learning framework to study algorithmic collusion in market making. The framework includes a self-interested market maker (Agent~A), which is trained in an uncertain environment shaped by an adversary, and three bottom-layer competitors: the self-interested Agent~B1 (whose objective is to maximize its own PnL), the competitive Agent~B2 (whose objective is to minimize the PnL of its opponent), and the hybrid Agent~B$^\star$, which can modulate between the behavior of the other two. To analyze how these agents shape the behavior of each other and affect market outcomes, we propose interaction-level metrics that quantify behavioral asymmetry and system-level dynamics, while providing signals potentially indicative of emergent interaction patterns. Experimental results show that Agent~B2 secures dominant performance in a zero-sum setting against B1, aggressively capturing order flow while tightening average spreads, thus improving market execution efficiency. In contrast, Agent~B$^\star$ exhibits a self-interested inclination when co-existing with other profit-seeking agents, securing dominant market share through adaptive quoting, yet exerting a milder adverse impact on the rewards of Agents~A and B1 compared to B2. These findings suggest that adaptive incentive control supports more sustainable strategic co-existence in heterogeneous agent environments and offers a structured lens for evaluating behavioral design in algorithmic trading systems.

Related papers

• Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets [74.91125572848439]
  We study two-sided agentic marketplaces where Assistant agents represent consumers and Service agents represent competing businesses.<n>This environment enables us to study key market dynamics: the utility agents achieve, behavioral biases, vulnerability to manipulation, and how search mechanisms shape market outcomes.<n>Our experiments show that frontier models can approach optimal welfare-- but only under ideal search conditions. Performance degrades sharply with scale, and all models exhibit severe first-proposal bias, creating 10-30x advantages for response speed over quality.
  arXiv  Detail & Related papers  (2025-10-27T18:35:59Z)
• AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning [78.5751183537704]
  AdvEvo-MARL is a co-evolutionary multi-agent reinforcement learning framework that internalizes safety into task agents.<n>Rather than relying on external guards, AdvEvo-MARL jointly optimize attackers and defenders.
  arXiv  Detail & Related papers  (2025-10-02T02:06:30Z)
• Can an Individual Manipulate the Collective Decisions of Multi-Agents? [53.01767232004823]
  M-Spoiler is a framework that simulates agent interactions within a multi-agent system to generate adversarial samples.<n>M-Spoiler introduces a stubborn agent that actively aids in optimizing adversarial samples.<n>Our findings confirm the risks posed by the knowledge of an individual agent in multi-agent systems.
  arXiv  Detail & Related papers  (2025-09-20T01:54:20Z)
• Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning [49.31650627835956]
  Partial agent failure becomes inevitable when systems scale up, making it crucial to identify the subset of agents whose compromise would most severely degrade overall performance.<n>In this paper, we study this Vulnerable Agent Identification (VAI) problem in large-scale multi-agent reinforcement learning (MARL)<n> Experiments show our method effectively identifies more vulnerable agents in large-scale MARL and the rule-based system, fooling system into worse failures, and learning a value function that reveals the vulnerability of each agent.
  arXiv  Detail & Related papers  (2025-09-18T16:03:50Z)
• Evaluating LLM Agent Collusion in Double Auctions [1.3194391758295114]
  We study the behavior of large language models (LLMs) acting as sellers in simulated double auction markets.<n>We find that direct seller communication increases collusive tendencies, the propensity to collude varies across models, and environmental pressures, such as oversight and urgency from authority figures, influence collusive behavior.
  arXiv  Detail & Related papers  (2025-07-02T07:06:49Z)
• DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning [84.22561239481901]
  We propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents. We evaluate DCIR in multiple environments including Multi-agent Particle, Google Research Football and StarCraft II Micromanagement.
  arXiv  Detail & Related papers  (2023-12-10T06:03:57Z)
• ProAgent: Building Proactive Cooperative Agents with Large Language Models [89.53040828210945]
  ProAgent is a novel framework that harnesses large language models to create proactive agents. ProAgent can analyze the present state, and infer the intentions of teammates from observations. ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various coordination scenarios.
  arXiv  Detail & Related papers  (2023-08-22T10:36:56Z)
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents [2.5204420653245245]
  Multi-Agent Reinforcement Learning (MARL) has been widely applied in many fields such as smart traffic and unmanned aerial vehicles. This work proposes a novel Robustness Testing framework for MARL that attacks states of Critical Agents.
  arXiv  Detail & Related papers  (2023-06-09T02:26:28Z)
• Artificial Intelligence and Dual Contract [2.1756081703276]
  We develop a model where two principals, each equipped with independent Q-learning algorithms, interact with a single agent. Our findings reveal that the strategic behavior of AI principals hinges crucially on the alignment of their profits.
  arXiv  Detail & Related papers  (2023-03-22T07:31:44Z)
• Multi-Issue Bargaining With Deep Reinforcement Learning [0.0]
  This paper evaluates the use of deep reinforcement learning in bargaining games. Two actor-critic networks were trained for the bidding and acceptance strategy. Neural agents learn to exploit time-based agents, achieving clear transitions in decision preference values. They also demonstrate adaptive behavior against different combinations of concession, discount factors, and behavior-based strategies.
  arXiv  Detail & Related papers  (2020-02-18T18:33:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.