Improving Generative Ad Text on Facebook using Reinforcement Learning

• URL: http://arxiv.org/abs/2507.21983v1
• Date: Tue, 29 Jul 2025 16:34:02 GMT
• Title: Improving Generative Ad Text on Facebook using Reinforcement Learning
• Authors: Daniel R. Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, Zheqing Zhu,
• Abstract summary: We present the first deployment of an RL-trained large language model for generative advertising on Facebook.<n>Our model, "AdLlama," powers an AI tool that helps advertisers create new variations of human-written ad text.<n>In a large-scale 10-week A/B test on Facebook, we find that AdLlama improves click-through rates by 6.7% compared to a supervised imitation model trained on curated ads.
• Score: 11.28110246872973
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative artificial intelligence (AI), in particular large language models (LLMs), is poised to drive transformative economic change. LLMs are pre-trained on vast text data to learn general language patterns, but a subsequent post-training phase is critical to align them for specific real-world tasks. Reinforcement learning (RL) is the leading post-training technique, yet its economic impact remains largely underexplored and unquantified. We examine this question through the lens of the first deployment of an RL-trained LLM for generative advertising on Facebook. Integrated into Meta's Text Generation feature, our model, "AdLlama," powers an AI tool that helps advertisers create new variations of human-written ad text. To train this model, we introduce reinforcement learning with performance feedback (RLPF), a post-training method that uses historical ad performance data as a reward signal. In a large-scale 10-week A/B test on Facebook spanning nearly 35,000 advertisers and 640,000 ad variations, we find that AdLlama improves click-through rates by 6.7% (p=0.0296) compared to a supervised imitation model trained on curated ads. This represents a substantial improvement in advertiser return on investment on Facebook. We also find that advertisers who used AdLlama generated more ad variations, indicating higher satisfaction with the model's outputs. To our knowledge, this is the largest study to date on the use of generative AI in an ecologically valid setting, offering an important data point quantifying the tangible impact of RL post-training. Furthermore, the results show that RLPF is a promising and generalizable approach for metric-driven post-training that bridges the gap between highly capable language models and tangible outcomes.

Related papers

• SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data [65.56911325914582]
  We propose Self-play Reinforcement Learning (SeRL) to bootstrap Large Language Models (LLMs) training with limited initial data.<n>The proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards.
  arXiv  Detail & Related papers  (2025-05-25T13:28:04Z)
• LLM Post-Training: A Deep Dive into Reasoning Large Language Models [131.10969986056]
  Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications.<n>Post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations.
  arXiv  Detail & Related papers  (2025-02-28T18:59:54Z)
• CTR-Driven Advertising Image Generation with Multimodal Large Language Models [53.40005544344148]
  We explore the use of Multimodal Large Language Models (MLLMs) for generating advertising images by optimizing for Click-Through Rate (CTR) as the primary objective.<n>To further improve the CTR of generated images, we propose a novel reward model to fine-tune pre-trained MLLMs through Reinforcement Learning (RL)<n>Our method achieves state-of-the-art performance in both online and offline metrics.
  arXiv  Detail & Related papers  (2025-02-05T09:06:02Z)
• Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learning [0.0]
  We propose a backdoor attack that focuses solely on data poisoning and a method of detection by dynamic systems and statistical analysis of the distribution of data.<n>Our aim is to examine the potential effects of large language models that use reinforcement learning systems for text production or speech recognition, finance, physics, or the ecosystem of contemporary artificial intelligence models.
  arXiv  Detail & Related papers  (2024-12-23T19:04:46Z)
• Self-Evolved Reward Learning for LLMs [45.6910747154447]
  Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences.<n>We propose Self-Evolved Reward Learning (SER), a novel approach where the RM generates additional training data to iteratively improve itself.<n>Our results demonstrate that even with limited human-annotated data, learning from self-feedback can robustly enhance RM performance.
  arXiv  Detail & Related papers  (2024-11-01T07:29:03Z)
• Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning [6.404122934568861]
  Supervised learning (SL) approaches have achieved impressive performance while utilizing significantly less training data compared to previous methods. We propose a novel approach that combines SL and RL techniques over the MiniWoB benchmark to leverage the strengths of both methods. Our experiments demonstrate that our approach outperforms previous SL methods on certain tasks using less data and narrows the performance gap with RL models.
  arXiv  Detail & Related papers  (2024-05-01T13:51:45Z)
• The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction [22.659005954676598]
  We show that it is possible to significantly improve the performance of Large Language Models by selectively removing higher-order components of their weight matrices. This simple intervention, which we call LAyer-SElective Rank reduction (LASER), can be done on a model after training has completed. We show extensive experiments demonstrating the generality of this finding across language models and datasets.
  arXiv  Detail & Related papers  (2023-12-21T03:51:08Z)
• Effective Large Language Model Adaptation for Improved Grounding and Citation Generation [48.07830615309543]
  This paper focuses on improving large language models (LLMs) by grounding their responses in retrieved passages and by providing citations. We propose a new framework, AGREE, that improves the grounding from a holistic perspective. Our framework tunes LLMs to selfground the claims in their responses and provide accurate citations to retrieved documents.
  arXiv  Detail & Related papers  (2023-11-16T03:22:25Z)
• A Simple yet Efficient Ensemble Approach for AI-generated Text Detection [0.5840089113969194]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in generating text that closely resembles human writing. It is essential to build automated approaches capable of distinguishing between artificially generated text and human-authored text. We propose a simple yet efficient solution by ensembling predictions from multiple constituent LLMs.
  arXiv  Detail & Related papers  (2023-11-06T13:11:02Z)
• The Web Can Be Your Oyster for Improving Large Language Models [98.72358969495835]
  Large language models (LLMs) encode a large amount of world knowledge. We consider augmenting LLMs with the large-scale web using search engine. We present a web-augmented LLM UNIWEB, which is trained over 16 knowledge-intensive tasks in a unified text-to-text format.
  arXiv  Detail & Related papers  (2023-05-18T14:20:32Z)
• A Survey of Large Language Models [81.06947636926638]
  Language modeling has been widely studied for language understanding and generation in the past two decades. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size.
  arXiv  Detail & Related papers  (2023-03-31T17:28:46Z)
• Ranking Creative Language Characteristics in Small Data Scenarios [52.00161818003478]
  We adapt the DirectRanker to provide a new deep model for ranking creative language with small data. Our experiments with sparse training data show that while the performance of standard neural ranking approaches collapses with small datasets, DirectRanker remains effective.
  arXiv  Detail & Related papers  (2020-10-23T18:57:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.