LLM Retrieval for Stable and Predictable Ad Recommendations

• URL: http://arxiv.org/abs/2605.21969v1
• Date: Thu, 21 May 2026 03:58:37 GMT
• Title: LLM Retrieval for Stable and Predictable Ad Recommendations
• Abstract summary: We introduce a new evaluation framework for quantifying stability and predictability of an ads recommender system.<n>We present an online validated semantic candidate generation framework powered by fine-tuned Large Language Models (LLMs)<n>This framework shows significant improvement along these metrics by fundamentally improving the semantic-awareness of the system.
• Score: 4.132360900725765
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional ads recommendation systems have primarily focused on optimizing for prediction accuracy of click or conversion events using canonical metrics such as recall or normalized discounted cumulative gain (NDCG). With the hyper-growth of ads inventory and liquidity with generative AI technologies, the prediction stability and predictability is becoming increasingly critical. Intuitively, prediction stability and predictability can be defined to quantify system robustness with respect to minor/noisy input (ads, creatives) perturbations, the lack of which could lead to advertiser perceivable problems such as repeatability, cold start and under-exploration. In this paper, we introduce a new evaluation framework for quantifying stability and predictability of an ads recommender system, and present an online validated semantic candidate generation framework powered by fine-tuned Large Language Models (LLMs) that showed significant improvement along these metrics by fundamentally improving the semantic-awareness of the system. The approach extracts hierarchical semantic attributes from ad creatives to obtain LLM representations, which serve as the foundation for graph-based expansion, ensuring the retrieved candidates encapsulate semantic variants of an ad, guaranteeing that small creative variants from the advertiser yield consistent and explainable delivery results to the user. We tested this LLM ads retrieval framework in a large-scale industrial ads recommendation system, demonstrating significant improvements across offline and online A/B experiments, showcasing gains in both predictability and traditional performance metrics. Although evaluated in the ads stack, this is a general framework that can be applied broadly to any large-scale recommendation and retrieval systems facing similar scaling and predictability challenges.

Related papers

• Aligning Data-Driven Predictors with Allocation: A Decision-Focused Approach to Survival Analysis [52.10284705210197]
  We show that machine learning predictors can yield arbitrarily poor outcomes when used for allocation.<n>We introduce a decision-focused learning approach based on optimizing normalized discounted cumulative gain (NDCG)<n>We propose a bootstrapping approach to optimize the NDCG of existing survival models.
  arXiv  Detail & Related papers  (2026-06-01T12:03:22Z)
• Specificity-aware reinforcement learning for fine-grained open-world classification [54.85385270439992]
  Classifying fine-grained visual concepts under open-world settings demands models to be both accurate and specific.<n>We propose a novel specificity-aware reinforcement learning framework, SpeciaRL, to fine-tune reasoning LMMs on fine-grained image classification.
  arXiv  Detail & Related papers  (2026-03-03T17:52:39Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• AWPO: Enhancing Tool-Use of Large Language Models through Explicit Integration of Reasoning Rewards [60.2998874976509]
  We propose advantage-weighted policy optimization (AWPO) to integrate explicit reasoning rewards to enhance tool-use capability.<n>AWPO incorporates variance-aware gating and difficulty-aware weighting to adaptively modulate advantages from reasoning signals.<n>Experiments demonstrate that AWPO achieves state-of-the-art performance across standard tool-use benchmarks.
  arXiv  Detail & Related papers  (2025-12-22T08:07:00Z)
• Breaking Determinism: Stochastic Modeling for Reliable Off-Policy Evaluation in Ad Auctions [16.315158617837646]
  This work contributes the first practical and validated framework for reliable Off-Policy Evaluation (OPE) in deterministic auction environments.<n>We introduce the first principled framework for OPE in deterministic auctions by repurposing the bid landscape model to approximate the propensity score.<n>We validate our approach on the AuctionNet simulation benchmark and against 2-weeks online A/B test from a large-scale industrial platform.
  arXiv  Detail & Related papers  (2025-12-03T01:37:42Z)
• EGA-V2: An End-to-end Generative Framework for Industrial Advertising [19.927005856735445]
  We introduce End-to-End Generative Advertising (EGA-V2), the first unified framework that holistically models user interests, point-of-interest (POI) and creative generation, ad allocation, and payment optimization.<n>Our results highlight its potential as a pioneering fully generative advertising solution, paving the way for next-generation industrial ad systems.
  arXiv  Detail & Related papers  (2025-05-23T06:55:02Z)
• A Novel Generative Model with Causality Constraint for Mitigating Biases in Recommender Systems [20.672668625179526]
  Latent confounding bias can obscure the true causal relationship between user feedback and item exposure.<n>We propose a novel generative framework called Latent Causality Constraints for Debiasing representation learning in Recommender Systems.
  arXiv  Detail & Related papers  (2025-05-22T14:09:39Z)
• A Survey of Direct Preference Optimization [103.59317151002693]
  Large Language Models (LLMs) have demonstrated unprecedented generative capabilities.<n>Their alignment with human values remains critical for ensuring helpful and harmless deployments.<n>Direct Preference Optimization (DPO) has recently gained prominence as a streamlined alternative.
  arXiv  Detail & Related papers  (2025-03-12T08:45:15Z)
• Beyond Self-Consistency: Loss-Balanced Perturbation-Based Regularization Improves Industrial-Scale Ads Ranking [17.264202675913012]
  Perturbation-based regularization techniques address many challenges in industrial-scale large models.<n>We propose Loss-Balanced Small Perturbation Regularization (LSPR) that can be used in potentially any deep learning model.<n>We have successfully demonstrated that both Self-Consistency Regularization approaches (SCR) and LSPR are scalable and can improve ads delivery systems.
  arXiv  Detail & Related papers  (2025-02-05T23:24:52Z)
• Unveiling Vulnerabilities of Contrastive Recommender Systems to Poisoning Attacks [48.911832772464145]
  Contrastive learning (CL) has recently gained prominence in the domain of recommender systems. This paper identifies a vulnerability of CL-based recommender systems that they are more susceptible to poisoning attacks aiming to promote individual items.
  arXiv  Detail & Related papers  (2023-11-30T04:25:28Z)
• Benchmarking PtO and PnO Methods in the Predictive Combinatorial Optimization Regime [59.27851754647913]
  Predictive optimization is the precise modeling of many real-world applications, including energy cost-aware scheduling and budget allocation on advertising. We develop a modular framework to benchmark 11 existing PtO/PnO methods on 8 problems, including a new industrial dataset for advertising. Our study shows that PnO approaches are better than PtO on 7 out of 8 benchmarks, but there is no silver bullet found for the specific design choices of PnO.
  arXiv  Detail & Related papers  (2023-11-13T13:19:34Z)
• COPR: Consistency-Oriented Pre-Ranking for Online Advertising [27.28920707332434]
  We introduce a consistency-oriented pre-ranking framework for online advertising. It employs a chunk-based sampling module and a plug-and-play rank alignment module to explicitly optimize consistency of ECPM-ranked results. When deployed in Taobao display advertising system, it achieves an improvement of up to +12.3% CTR and +5.6% RPM.
  arXiv  Detail & Related papers  (2023-06-06T09:08:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.