Dynamic Bidding Strategies with Multivariate Feedback Control for Multiple Goals in Display Advertising

• URL: http://arxiv.org/abs/2007.00426v1
• Date: Mon, 1 Jun 2020 15:49:08 GMT
• Title: Dynamic Bidding Strategies with Multivariate Feedback Control for Multiple Goals in Display Advertising
• Abstract summary: Real-Time Bidding (RTB) display advertising is a method for purchasing display advertising inventory in auctions that occur within milliseconds. The performance of RTB campaigns is generally measured with a series of Key Performance Indicators (KPIs) Here we present an approach to simultaneously controlling multiple traffic with a PID-based feedback-control system. In both offline ad bidding simulations and testing on live traffic, our methods proved to be effective in simultaneously controlling multiple traffic, and bringing them toward their respective goals.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-Time Bidding (RTB) display advertising is a method for purchasing display advertising inventory in auctions that occur within milliseconds. The performance of RTB campaigns is generally measured with a series of Key Performance Indicators (KPIs) - measurements used to ensure that the campaign is cost-effective and that it is purchasing valuable inventory. While an RTB campaign should ideally meet all KPIs, simultaneous improvement tends to be very challenging, as an improvement to any one KPI risks a detrimental effect toward the others. Here we present an approach to simultaneously controlling multiple KPIs with a PID-based feedback-control system. This method generates a control score for each KPI, based on both the output of a PID controller module and a metric that quantifies the importance of each KPI for internal business needs. On regular intervals, this algorithm - Sequential Control - will choose the KPI with the greatest overall need for improvement. In this way, our algorithm is able to continually seek the greatest marginal improvements to its current state. Multiple methods of control can be associated with each KPI, and can be triggered either simultaneously or chosen stochastically, in order to avoid local optima. In both offline ad bidding simulations and testing on live traffic, our methods proved to be effective in simultaneously controlling multiple KPIs, and bringing them toward their respective goals.

Related papers

• Beyond Single-Episode Optimization: Sliding-Window Aware Generative Auto-Bidding for Long-Term Advertising Effectiveness [8.51409961240706]
  Auto-bidding systems optimize bids to maximize value under efficiency constraints such as Cost-Per-Action (CPA)<n>We propose SWAG-Bid, a hierarchical framework decomposing this challenge into episode-level planning and step-level execution.<n>Experiments on AuctionNet-Sparse and online A/B tests on show that SWAG-Bid achieves competitive constraint satisfaction and value acquisition under sliding-window evaluation.
  arXiv  Detail & Related papers  (2026-07-28T03:21:13Z)
• GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization [54.596224644751565]
  Post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities.<n>To address this, existing methods such as Group reward-Decoupled Policy Optimization (GDPO) decompose the overall score into independent reward groups, then compute the RL loss separately within each group.<n>We propose Group-Dynamic reward-Decoupled Policy Optimization (GD$2$PO), which employs a conflict-aware filtering mechanism to mask out rollouts suffering from severe reward-wise disagreement.
  arXiv  Detail & Related papers  (2026-06-15T14:19:28Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewards [36.029144318322686]
  We model ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards.<n>For efficient estimation, we propose a two-stage maximum likelihood estimator combined with data-splitting strategies.<n>We design a reinforcement learning algorithm to derive efficient personalized bidding strategies.
  arXiv  Detail & Related papers  (2025-10-22T22:08:36Z)
• Autobidding Arena: unified evaluation of the classical and RL-based autobidding algorithms [71.47275796833235]
  We present a standardized and transparent evaluation protocol for comparing classical and reinforcement learning autobidding algorithms.<n>We utilize the most recent open-source environment developed in the industry, which accurately emulates the bidding process.
  arXiv  Detail & Related papers  (2025-10-22T08:27:56Z)
• Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems [54.709976343045824]
  Current offline reinforcement learning (RL) methods face substantial challenges when applied to sparse advertising scenarios.<n>We propose MTORL, a novel multi-task offline RL model that targets two key objectives.<n>We employ multi-task learning to decode actions and rewards, simultaneously addressing channel recommendation and budget allocation.
  arXiv  Detail & Related papers  (2025-06-29T05:05:13Z)
• Real Time Self-Tuning Adaptive Controllers on Temperature Control Loops using Event-based Game Theory [3.156133122658661]
  This paper presents a novel method for enhancing the adaptability of Proportional-Integral-Derivative (PID) controllers in industrial systems using event-based dynamic game theory.<n>In contrast to conventional self-learning approaches, our proposed framework offers an event-driven control strategy and game-theoretic learning algorithms.
  arXiv  Detail & Related papers  (2025-06-16T07:19:46Z)
• Review, Refine, Repeat: Understanding Iterative Decoding of AI Agents with Dynamic Evaluation and Selection [71.92083784393418]
  Inference-time methods such as Best-of-N (BON) sampling offer a simple yet effective alternative to improve performance. We propose Iterative Agent Decoding (IAD) which combines iterative refinement with dynamic candidate evaluation and selection guided by a verifier.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Procurement Auctions via Approximately Optimal Submodular Optimization [53.93943270902349]
  We study procurement auctions, where an auctioneer seeks to acquire services from strategic sellers with private costs. Our goal is to design computationally efficient auctions that maximize the difference between the quality of the acquired services and the total cost of the sellers.
  arXiv  Detail & Related papers  (2024-11-20T18:06:55Z)
• Multi-Scenario Combination Based on Multi-Agent Reinforcement Learning to Optimize the Advertising Recommendation System [38.501423778989704]
  We introduce the Multi-Agent Recurrent Deterministic Policy Gradient (MARDPG) algorithm, which aligns different scenarios under a shared objective. Our results show marked improvements in metrics such as click-through rate (CTR), conversion rate, and total sales.
  arXiv  Detail & Related papers  (2024-07-03T02:33:20Z)
• When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL [37.58940726230092]
  Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP) We formalize an RL framework, Time-adaptive Control & Sensing (TaCoS), that tackles this challenge. We demonstrate that state-of-the-art RL algorithms trained on TaCoS drastically reduce the interaction amount over their discrete-time counterpart.
  arXiv  Detail & Related papers  (2024-06-03T09:57:18Z)
• Distributed-Training-and-Execution Multi-Agent Reinforcement Learning for Power Control in HetNet [48.96004919910818]
  We propose a multi-agent deep reinforcement learning (MADRL) based power control scheme for the HetNet. To promote cooperation among agents, we develop a penalty-based Q learning (PQL) algorithm for MADRL systems. In this way, an agent's policy can be learned by other agents more easily, resulting in a more efficient collaboration process.
  arXiv  Detail & Related papers  (2022-12-15T17:01:56Z)
• Policy Search for Model Predictive Control with Application to Agile Drone Flight [56.24908013905407]
  We propose a policy-search-for-model-predictive-control framework for MPC. Specifically, we formulate the MPC as a parameterized controller, where the hard-to-optimize decision variables are represented as high-level policies. Experiments show that our controller achieves robust and real-time control performance in both simulation and the real world.
  arXiv  Detail & Related papers  (2021-12-07T17:39:24Z)
• The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games [67.47961797770249]
  Multi-Agent PPO (MAPPO) is a multi-agent PPO variant which adopts a centralized value function. We show that MAPPO achieves performance comparable to the state-of-the-art in three popular multi-agent testbeds.
  arXiv  Detail & Related papers  (2021-03-02T18:59:56Z)
• Learning over no-Preferred and Preferred Sequence of items for Robust Recommendation [66.8722561224499]
  We propose a theoretically founded sequential strategy for training large-scale Recommender Systems (RS) over implicit feedback. We present two variants of this strategy where model parameters are updated using either the momentum method or a gradient-based approach.
  arXiv  Detail & Related papers  (2020-12-12T22:10:15Z)
• Deep Controlled Learning for Inventory Control [0.0]
  Controlled Deep Learning (DCL) is a new DRL framework based on approximate policy specifically designed to tackle inventory problems. DCL outperforms existing state-of-the-art iterations in lost sales inventory control, perishable inventory systems, and inventory systems with random lead times. These substantial performance and robustness improvements pave the way for the effective application of tailored DRL algorithms to inventory management problems.
  arXiv  Detail & Related papers  (2020-11-30T18:53:08Z)
• Optimal Bidding Strategy without Exploration in Real-time Bidding [14.035270361462576]
  maximizing utility with a budget constraint is the primary goal for advertisers in real-time bidding (RTB) systems. Previous works ignore the losing auctions to alleviate the difficulty with censored states. We propose a novel practical framework using the maximum entropy principle to imitate the behavior of the true distribution observed in real-time traffic.
  arXiv  Detail & Related papers  (2020-03-31T20:43:28Z)
• MoTiAC: Multi-Objective Actor-Critics for Real-Time Bidding [47.555870679348416]
  We propose a Multi-ecTive Actor-Critics algorithm named MoTiAC for the problem of bidding optimization with various goals. Unlike previous RL models, the proposed MoTiAC can simultaneously fulfill multi-objective tasks in complicated bidding environments.
  arXiv  Detail & Related papers  (2020-02-18T07:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.