Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies

• URL: http://arxiv.org/abs/2601.02754v1
• Date: Tue, 06 Jan 2026 06:42:25 GMT
• Title: Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies
• Abstract summary: We propose QGA, a Q-value regularized Generative Auto-bidding method.<n>In large-scale real-world A/B testing, QGA achieves a 3.27% increase in Ad GMV and a 2.49% improvement in Ad ROI.
• Score: 27.96571937113551
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: With the rapid development of e-commerce, auto-bidding has become a key asset in optimizing advertising performance under diverse advertiser environments. The current approaches focus on reinforcement learning (RL) and generative models. These efforts imitate offline historical behaviors by utilizing a complex structure with expensive hyperparameter tuning. The suboptimal trajectories further exacerbate the difficulty of policy learning. To address these challenges, we proposes QGA, a novel Q-value regularized Generative Auto-bidding method. In QGA, we propose to plug a Q-value regularization with double Q-learning strategy into the Decision Transformer backbone. This design enables joint optimization of policy imitation and action-value maximization, allowing the learned bidding policy to both leverage experience from the dataset and alleviate the adverse impact of the suboptimal trajectories. Furthermore, to safely explore the policy space beyond the data distribution, we propose a Q-value guided dual-exploration mechanism, in which the DT model is conditioned on multiple return-to-go targets and locally perturbed actions. This entire exploration process is dynamically guided by the aforementioned Q-value module, which provides principled evaluation for each candidate action. Experiments on public benchmarks and simulation environments demonstrate that QGA consistently achieves superior or highly competitive results compared to existing alternatives. Notably, in large-scale real-world A/B testing, QGA achieves a 3.27% increase in Ad GMV and a 2.49% improvement in Ad ROI.

Related papers

• DEGR: Dual Exploration-Driven Generative Re-Ranking for Adaptive Cross-Request Context Bridging [7.797900623517651]
  We propose a Dual Exploration-Driven Generative Re-Ranking (DEGR) method for industrial recommendation systems.<n>In experiments, DEGR outperforms SOTA methods, achieving improvements of up to 1.22% UCTR and 0.20% PV in the JD E-commerce recommendation system.
  arXiv  Detail & Related papers  (2026-08-05T13:14:23Z)
• AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization [23.02295184305076]
  Auto-bidding plays an essential role in online advertising, automatically adjusting bids for advertisers to optimize their commercial goals.<n>We propose AIGB-R1, a hierarchical self-evolving auto-bidding framework aiming to enhance AI-Generated Bidding via Reasoning capabilities.<n>We adopt a two-stage pipeline of offline pre-training and post-training alignment, and build an interactive bidding simulation environment for strategy rollout.
  arXiv  Detail & Related papers  (2026-07-19T14:59:12Z)
• Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning [76.93011742289768]
  Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories.<n>This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills.<n>We propose Strategy-Guided Policy Optimization, which replaces instance-level trajectory imitation with reusable strategy distillation.
  arXiv  Detail & Related papers  (2026-06-23T02:14:12Z)
• HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising [4.325040293861279]
  HOBA is a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales.<n> Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-17T15:00:34Z)
• Aligning Flow Map Policies with Optimal Q-Guidance [50.514994916864275]
  Flow map policies are designed for fast action generation by learning to take arbitrary-size jumps.<n>We derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint.<n>FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.
  arXiv  Detail & Related papers  (2026-05-12T17:12:29Z)
• AIGQ: An End-to-End Hybrid Generative Architecture for E-commerce Query Recommendation [9.218475190088734]
  Pre-search query recommendation, widely known as HintQ on Taobao's homepage, plays a vital role in intent capture and demand discovery.<n>To overcome these challenges, we propose AIGQ, the first end-to-end generative framework for HintQ scenario.
  arXiv  Detail & Related papers  (2026-03-20T07:27:59Z)
• Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization [38.863014369090074]
  We introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization.<n>Our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on GUI navigation and grounding benchmarks.
  arXiv  Detail & Related papers  (2026-02-14T07:44:47Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• Generative Actor Critic [74.04971271003869]
  Generative Actor Critic (GAC) is a novel framework that decouples sequential decision-making by reframing textitpolicy evaluation as learning a generative model of the joint distribution over trajectories and returns.<n>Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-12-25T06:31:11Z)
• ORPR: An OR-Guided Pretrain-then-Reinforce Learning Model for Inventory Management [9.138155308817215]
  "Pretrain-then-Reinforce" approach reconciles AI's adaptive perception with Operations Research's structural rigor.<n>We show that a lightweight, domain-informed model can deliver state-of-the-art performance and robust transferability when guided by structured OR logic.
  arXiv  Detail & Related papers  (2025-12-22T03:39:43Z)
• Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving [54.46325690390831]
  We propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment.<n>MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine.<n>MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes.
  arXiv  Detail & Related papers  (2025-11-26T17:01:41Z)
• Automatic Policy Search using Population-Based Hyper-heuristics for the Integrated Procurement and Perishable Inventory Problem [3.4792548480344245]
  We develop a discrete-event simulation environment to compare two optimization strategies for this multi-item, multi-supplier problem.<n>Results from twelve distinct instances demonstrate that the hyper-heuristic framework consistently identifies superior policies.
  arXiv  Detail & Related papers  (2025-11-02T01:27:52Z)
• Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition [52.232968183793986]
  General Policy Composition (GPC) is a training-free method that enhances performance by combining the distributional scores of multiple pre-trained policies.<n>GPC consistently improves performance and adaptability across a diverse set of tasks.
  arXiv  Detail & Related papers  (2025-10-01T16:05:53Z)
• Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search [24.02739832976663]
  Auto-bidding serves as a critical tool for advertisers to improve their performance.<n>Recent progress has demonstrated that AI-Generated Bidding (AIGB) achieves superior performance compared to typical offline reinforcement learning (RL)-based auto-bidding methods.<n>We propose AIGB-Pearl, a novel method that integrates generative planning and policy optimization.
  arXiv  Detail & Related papers  (2025-09-19T12:30:26Z)
• Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimization [5.460538555236247]
  We propose GRAD (Generative Reward-driven Ad-bidding with Mixture-of-Experts), a scalable foundation model for auto-bidding.<n>We show that GRAD significantly enhances platform revenue, highlighting its effectiveness in addressing the evolving and diverse requirements of modern advertisers.
  arXiv  Detail & Related papers  (2025-08-04T02:46:18Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Q-value Regularized Transformer for Offline Reinforcement Learning [70.13643741130899]
  We propose a Q-value regularized Transformer (QT) to enhance the state-of-the-art in offline reinforcement learning (RL) QT learns an action-value function and integrates a term maximizing action-values into the training loss of Conditional Sequence Modeling (CSM) Empirical evaluations on D4RL benchmark datasets demonstrate the superiority of QT over traditional DP and CSM methods.
  arXiv  Detail & Related papers  (2024-05-27T12:12:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.