OOM-RL: Out-of-Money Reinforcement Learning Market-Driven Alignment for LLM-Based Multi-Agent Systems

• URL: http://arxiv.org/abs/2604.11477v1
• Date: Mon, 13 Apr 2026 13:45:42 GMT
• Title: OOM-RL: Out-of-Money Reinforcement Learning Market-Driven Alignment for LLM-Based Multi-Agent Systems
• Abstract summary: We introduce an objective alignment paradigm: textbfOut-of-Money Reinforcement Learning (OOM-RL).<n>Our study chronicles the system's evolution from a high-turnover, sycophantic baseline to a robust, liquidity-aware architecture.<n>We conclude that substituting subjective human preference with rigorous economic penalties provides a robust methodology for aligning autonomous agents in high-stakes, real-world environments.
• Score: 4.970665847294818
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The alignment of Multi-Agent Systems (MAS) for autonomous software engineering is constrained by evaluator epistemic uncertainty. Current paradigms, such as Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF), frequently induce model sycophancy, while execution-based environments suffer from adversarial "Test Evasion" by unconstrained agents. In this paper, we introduce an objective alignment paradigm: \textbf{Out-of-Money Reinforcement Learning (OOM-RL)}. By deploying agents into the non-stationary, high-friction reality of live financial markets, we utilize critical capital depletion as an un-hackable negative gradient. Our longitudinal 20-month empirical study (July 2024 -- February 2026) chronicles the system's evolution from a high-turnover, sycophantic baseline to a robust, liquidity-aware architecture. We demonstrate that the undeniable ontological consequences of financial loss forced the MAS to abandon overfitted hallucinations in favor of the \textbf{Strict Test-Driven Agentic Workflow (STDAW)}, which enforces a Byzantine-inspired uni-directional state lock (RO-Lock) anchored to a deterministically verified $\geq 95\%$ code coverage constraint matrix. Our results show that while early iterations suffered severe execution decay, the final OOM-RL-aligned system achieved a stable equilibrium with an annualized Sharpe ratio of 2.06 in its mature phase. We conclude that substituting subjective human preference with rigorous economic penalties provides a robust methodology for aligning autonomous agents in high-stakes, real-world environments, laying the groundwork for generalized paradigms where computational billing acts as an objective physical constraint

Related papers

• From Viral to Void: Multi-Dimensional Behavioral and Contractual Analysis for Rug Pull Identification [2.7875224007687827]
  Rug Pull scams are characterized by extremely low deployment costs, covert execution, rapid fund transfers, and high detection difficulty.<n>This paper develops a multi-dimensional feature system based on dimensions such as malicious smart contract design, on-chain transaction anomalies, liquidity manipulation, and social media disclosures.<n> Experiments show that the model achieves an accuracy of 0.927, an F1 score of 0.787, and an AUC-ROC of 0.952 on the test set, outperforming traditional methods.
  arXiv  Detail & Related papers  (2026-08-03T02:34:24Z)
• ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples [65.74946647574696]
  Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs)<n>In this work, we investigate a critical source of this instability: over-optimization.<n>We propose ARMOR, a framework that shifts the paradigm from passive penalty to active sample stabilization.
  arXiv  Detail & Related papers  (2026-07-11T21:22:46Z)
• Semantic Pareto-DQN: A Multi-Objective Reinforcement Learning Framework for Financial Anomaly Detection [0.2864713389096699]
  We propose a multi-objective reinforcement learning framework to overcome class imbalance in financial anomaly detection.<n>Our approach synthesizes heterogeneous transaction features into cohesive natural-narratives, encoded by large language models.<n>It achieves superior minority-class recall compared to scalarized baselines, providing an alternative to trade bounded operational friction for financial anomaly discovery.
  arXiv  Detail & Related papers  (2026-07-10T17:39:00Z)
• Provably Secure Agent Guardrail [89.79561918065122]
  Existing defense architectures rely on empirical semantic guardrails and probabilistic large model adjudicators.<n>This paper proposes a new security paradigm for agents based on the fundamental limitations of logical reasoning.
  arXiv  Detail & Related papers  (2026-05-28T02:12:41Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• A Self-Healing Framework for Reliable LLM-Based Autonomous Agents [0.0]
  This paper proposes a reliability-aware self-healing framework for LLM-based software agents.<n>The proposed framework was implemented in a multi-agent workflow environment and evaluated using real-world task scenarios.
  arXiv  Detail & Related papers  (2026-05-07T13:10:41Z)
• Bounded by Risk, Not Capability: Quantifying AI Occupational Substitution Rates via a Tech-Risk Dual-Factor Model [0.0]
  We argue occupations are not eradicated instantaneously, but gradually encroached upon via atomic actions.<n>By deconstructing 923 occupations into 2,087 Detailed Work Activities (DWAs), we utilize a multi-agent LLM ensemble to score both technical feasibility and business risk.
  arXiv  Detail & Related papers  (2026-04-06T06:21:08Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• A Blueprint for Self-Evolving Coding Agents in Vehicle Aerodynamic Drag Prediction [9.020568132598827]
  We present a contract-centric blueprint for self-evolving coding agents that discover executable surrogate pipelines for predicting drag coefficient $C_d$ under industrial constraints.<n>The method formulates surrogate discovery as constrained optimization over programs, not static model instances.<n>A hard evaluation contract enforces leakage prevention, deterministic replay, multi-seed, robustness and resource budgets before any candidate is admitted.
  arXiv  Detail & Related papers  (2026-03-23T08:36:57Z)
• Socially-Weighted Alignment: A Game-Theoretic Framework for Multi-Agent LLM Systems [17.658093330392052]
  We propose a game-theoretic framework that modifies inference-time decision making by interpolating between an agent's private objective and an estimate of group welfare.<n>We show that SWA induces a critical threshold $*=(n-)/(n-1)$ above which agents no longer have marginal incentive to increase demand under overload.
  arXiv  Detail & Related papers  (2026-02-16T05:17:58Z)
• Unifying Stable Optimization and Reference Regularization in RLHF [64.16830602324345]
  This paper introduces a unified regularization approach that balances objectives of preventing reward hacking and maintaining stable policy updates.<n>Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity.
  arXiv  Detail & Related papers  (2026-02-12T03:31:19Z)
• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling [49.41422138354821]
  We propose a principled reward modeling framework that integrates non-negative factor analysis into the Bradley-Terry preference model.<n>BNRM represents rewards through a sparse, non-negative latent factor generative process.<n>We show that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.
  arXiv  Detail & Related papers  (2026-02-11T08:14:11Z)
• The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agents [37.75212140218036]
  We formalize the Loss-of-Control risk and identify the previously underexamined Intrinsic Value Misalignment (Intrinsic VM)<n>We then introduce IMPRESS, a scenario-driven framework for systematically assessing this risk.<n>We evaluate Intrinsic VM on 21 state-of-the-art LLM agents and find that it is a common and broadly observed safety risk across models.
  arXiv  Detail & Related papers  (2026-01-24T07:09:50Z)
• Optimistic TEE-Rollups: A Hybrid Architecture for Scalable and Verifiable Generative AI Inference on Blockchain [4.254924788681319]
  We introduce Optimistic TEE-Rollups (OTR), a hybrid verification protocol that harmonizes constraints.<n>OTR achieves 99% of the throughput of centralized baselines with a marginal cost overhead of $0.07 per query.
  arXiv  Detail & Related papers  (2025-12-23T09:16:41Z)
• Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking [78.69179041551014]
  We propose an information-theoretic reward modeling framework based on the Information Bottleneck principle.<n>We show that InfoRM filters out preference-irrelevant information to alleviate reward misgeneralization.<n>We also introduce IBL, a distribution-level regularization that penalizes such deviations, effectively expanding the optimization landscape.
  arXiv  Detail & Related papers  (2025-10-15T15:51:59Z)
• One Token to Fool LLM-as-a-Judge [52.45386385722788]
  Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models.<n>We uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking.
  arXiv  Detail & Related papers  (2025-07-11T17:55:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.