HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising

• URL: http://arxiv.org/abs/2607.24779v1
• Date: Wed, 17 Jun 2026 15:00:34 GMT
• Title: HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
• Abstract summary: HOBA is a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales.<n> Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines.
• Score: 4.325040293861279
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online advertising bidding systems typically deploy multiple offline-trained expert models (e.g., PID controllers, model predictive control, offline RL policies) but face two critical limitations: lack of online adaptability to non-stationary auction markets, and reliance on costly manual tuning of hyperparameters such as bid bounds and budget pacing constraints. We propose HOBA (Hierarchical On-policy Bidding Agents), a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales. At the high level, a large language model infers hyperparameters from contextual signals through a Think-Act-Observe-Reflect loop with historical experience retrieval. At the mid level, a SARSA agent dynamically selects among expert models, incorporating causal adjustment to eliminate selection bias. At the low level, a dynamic expert pool (PID, MPC, IQL, Decision Transformer) executes bids under high-level constraints. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability. Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines. In a large-scale online deployment, HOBA delivered substantial business value, achieving a +3.6\% increase in target cost, proving the effectiveness of our hierarchical multi-agent bidding paradigm.

Related papers

• AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization [23.02295184305076]
  Auto-bidding plays an essential role in online advertising, automatically adjusting bids for advertisers to optimize their commercial goals.<n>We propose AIGB-R1, a hierarchical self-evolving auto-bidding framework aiming to enhance AI-Generated Bidding via Reasoning capabilities.<n>We adopt a two-stage pipeline of offline pre-training and post-training alignment, and build an interactive bidding simulation environment for strategy rollout.
  arXiv  Detail & Related papers  (2026-07-19T14:59:12Z)
• Selective Ensemble Based on Preference-Directed Multi-Objective Bandits [90.75513823660775]
  We formalize a sequential decision problem under partially specified linear preferences.<n>We then propose a preference-directed upper confidence bound (PrefUCB) algorithm.<n> Experiments on large pre-trained model selective ensemble tasks and online asset allocation under institutional mandates validate our method.
  arXiv  Detail & Related papers  (2026-06-20T07:52:46Z)
• DRIVE: Distributional and Retrieval-Augmented Bidding with Value Evaluation [39.181584470241354]
  offline reinforcement learning and Transformer-based sequence modeling have shown promise for learning bidding policies from logged data.<n>We propose DRIVE, a unified Transformer-based framework that decouples candidate action generation from decision making for offline auto-bidding.<n>Experiments on AuctionNet and additional offline reinforcement learning benchmarks demonstrate that DRIVE consistently improves bidding performance and generalizes well across multiple Transformer-based methods.
  arXiv  Detail & Related papers  (2026-06-12T07:21:07Z)
• AIPO: Learning to Reason from Active Interaction [54.10819421625103]
  AIPO enables the policy model to proactively consult three functional collaborative agents when encountering reasoning bottlenecks.<n>AIPO consistently improves reasoning performance, generalizes robustly across different policy models and RLVR algorithms, and effectively expands the reasoning capability boundary of the policy model.
  arXiv  Detail & Related papers  (2026-05-08T19:06:55Z)
• AHBid: An Adaptable Hierarchical Bidding Framework for Cross-Channel Advertising [8.53485049764747]
  AHBid is an Adaptable Hierarchical Bidding framework that integrates generative planning with real-time control.<n>Experiments conducted on large-scale offline datasets and through online A/B tests demonstrate the effectiveness of AHBid.
  arXiv  Detail & Related papers  (2026-02-26T06:07:28Z)
• SEGB: Self-Evolved Generative Bidding with Local Autoregressive Diffusion [9.051746879211764]
  Self-Evolved Generative Bidding (SEGB) is a framework that plans proactively and refines itself entirely offline.<n>SEGB first synthesizes plausible short-horizon future states to guide each bid, providing the agent with crucial, dynamic foresight.<n>It then performs value-guided policy refinement to iteratively discover superior strategies without any external intervention.
  arXiv  Detail & Related papers  (2025-12-31T09:05:59Z)
• Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems [54.916243942641444]
  Large language models (LLMs) are emerging as key enablers of automation in domains such as telecommunications.<n>We study an edge-cloud-expert cascaded LLM-based knowledge system that supports decision-making through a question-and-answer pipeline.
  arXiv  Detail & Related papers  (2025-12-23T03:10:09Z)
• Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs [69.2486294522259]
  BaRP is a Bandit Routing-feedback with Preferences approach that trains under the same partial-feedback restriction as deployment.<n> Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt.
  arXiv  Detail & Related papers  (2025-10-08T18:24:59Z)
• Steerable Adversarial Scenario Generation through Test-Time Preference Alignment [58.37104890690234]
  Adversarial scenario generation is a cost-effective approach for safety assessment of autonomous driving systems.<n>We introduce a new framework named textbfSteerable textbfAdversarial scenario textbfGEnerator (SAGE)<n>SAGE enables fine-grained test-time control over the trade-off between adversariality and realism without any retraining.
  arXiv  Detail & Related papers  (2025-09-24T13:27:35Z)
• Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimization [5.460538555236247]
  We propose GRAD (Generative Reward-driven Ad-bidding with Mixture-of-Experts), a scalable foundation model for auto-bidding.<n>We show that GRAD significantly enhances platform revenue, highlighting its effectiveness in addressing the evolving and diverse requirements of modern advertisers.
  arXiv  Detail & Related papers  (2025-08-04T02:46:18Z)
• Nash Equilibrium Constrained Auto-bidding With Bi-level Reinforcement Learning [64.2367385090879]
  We propose a new formulation of the auto-bidding problem from the platform's perspective.<n>It aims to maximize the social welfare of all advertisers under the $epsilon$-NE constraint.<n>The NCB problem presents significant challenges due to its constrained bi-level structure and the typically large number of advertisers involved.
  arXiv  Detail & Related papers  (2025-03-13T12:25:36Z)
• Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification [74.10976684469435]
  offline reinforcement learning (RL) algorithms can be transferred to multi-agent settings directly. We propose a simple yet effective method, Offline Multi-Agent RL with Actor Rectification (OMAR), to tackle this critical challenge. OMAR significantly outperforms strong baselines with state-of-the-art performance in multi-agent continuous control benchmarks.
  arXiv  Detail & Related papers  (2021-11-22T13:27:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.