Experimentation Accelerator: Interpretable Insights and Creative Recommendations for A/B Testing with Content-Aware ranking

• URL: http://arxiv.org/abs/2602.13852v1
• Date: Sat, 14 Feb 2026 19:22:58 GMT
• Title: Experimentation Accelerator: Interpretable Insights and Creative Recommendations for A/B Testing with Content-Aware ranking
• Abstract summary: We present a unified framework to prioritize which variants to test, (ii) explain why winners win, and (iii) surface targeted opportunities for new, higher-potential variants.<n>These components have been built into a real Adobe product, called textitExperimentation Accelerator, to provide AI-based insights and opportunities to scale experimentation for customers.
• Score: 28.316205612655228
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern online experimentation faces two bottlenecks: scarce traffic forces tough choices on which variants to test, and post-hoc insight extraction is manual, inconsistent, and often content-agnostic. Meanwhile, organizations underuse historical A/B results and rich content embeddings that could guide prioritization and creative iteration. We present a unified framework to (i) prioritize which variants to test, (ii) explain why winners win, and (iii) surface targeted opportunities for new, higher-potential variants. Leveraging treatment embeddings and historical outcomes, we train a CTR ranking model with fixed effects for contextual shifts that scores candidates while balancing value and content diversity. For better interpretability and understanding, we project treatments onto curated semantic marketing attributes and re-express the ranker in this space via a sign-consistent, sparse constrained Lasso, yielding per-attribute coefficients and signed contributions for visual explanations, top-k drivers, and natural-language insights. We then compute an opportunity index combining attribute importance (from the ranker) with under-expression in the current experiment to flag missing, high-impact attributes. Finally, LLMs translate ranked opportunities into concrete creative suggestions and estimate both learning and conversion potential, enabling faster, more informative, and more efficient test cycles. These components have been built into a real Adobe product, called \textit{Experimentation Accelerator}, to provide AI-based insights and opportunities to scale experimentation for customers. We provide an evaluation of the performance of the proposed framework on some real-world experiments by Adobe business customers that validate the high quality of the generation pipeline.

Related papers

• Optimizing In-Context Demonstrations for LLM-based Automated Grading [31.353360036776976]
  GUIDE (Grading Using Iteratively Designed Exemplars) is a framework that reframes exemplar selection and refinement as a boundary-focused optimization problem.<n>We show that GUIDE significantly outperforms standard retrieval baselines in experiments in physics, chemistry, and pedagogical content knowledge.
  arXiv  Detail & Related papers  (2026-02-28T04:52:38Z)
• LLM-Powered Nuanced Video Attribute Annotation for Enhanced Recommendations [3.461479036524523]
  This paper presents a case study on deploying Large Language Models (LLMs) as an advanced "annotation" mechanism to achieve nuanced content understanding.<n>The "LLM-as-annotators" approach addresses these by significantly shortening development times and enabling the annotation of subtle attributes.
  arXiv  Detail & Related papers  (2025-10-08T05:17:17Z)
• Reflect then Learn: Active Prompting for Information Extraction Guided by Introspective Confusion [41.79586757544166]
  Large Language Models (LLMs) show remarkable potential for few-shot information extraction (IE)<n> Conventional selection strategies often fail to provide informative guidance, as they overlook a key source of model fallibility.<n>We introduce Active Prompting for Information Extraction (APIE), a novel active prompting framework guided by a principle we term introspective confusion.
  arXiv  Detail & Related papers  (2025-08-10T02:27:41Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• A Systematic Examination of Preference Learning through the Lens of Instruction-Following [83.71180850955679]
  We use a novel synthetic data generation pipeline to generate 48,000 instruction unique-following prompts.<n>With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS)<n>Experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements.<n>High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance.
  arXiv  Detail & Related papers  (2024-12-18T15:38:39Z)
• A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems [67.52782366565658]
  State-of-the-art recommender systems (RSs) depend on categorical features, which ecoded by embedding vectors, resulting in excessively large embedding tables.<n>Despite the prosperity of lightweight embedding-based RSs, a wide diversity is seen in evaluation protocols.<n>This study investigates various LERS' performance, efficiency, and cross-task transferability via a thorough benchmarking process.
  arXiv  Detail & Related papers  (2024-06-25T07:45:00Z)
• LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments [2.2021543101231167]
  Modern media firms require automated and efficient methods to identify content that is most engaging and appealing to users. We first investigate the ability of three pure-LLM approaches to identify the catchiest headline: prompt-based methods, embedding-based methods, and fine-tuned open-source LLMs. We then introduce the LLM-Assisted Online Learning Algorithm (LOLA), a novel framework that integrates Large Language Models (LLMs) with adaptive experimentation to optimize content delivery.
  arXiv  Detail & Related papers  (2024-06-03T07:56:58Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Meta-Wrapper: Differentiable Wrapping Operator for User Interest Selection in CTR Prediction [97.99938802797377]
  Click-through rate (CTR) prediction, whose goal is to predict the probability of the user to click on an item, has become increasingly significant in recommender systems. Recent deep learning models with the ability to automatically extract the user interest from his/her behaviors have achieved great success. We propose a novel approach under the framework of the wrapper method, which is named Meta-Wrapper.
  arXiv  Detail & Related papers  (2022-06-28T03:28:15Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Generative Counterfactuals for Neural Networks via Attribute-Informed Perturbation [51.29486247405601]
  We design a framework to generate counterfactuals for raw data instances with the proposed Attribute-Informed Perturbation (AIP) By utilizing generative models conditioned with different attributes, counterfactuals with desired labels can be obtained effectively and efficiently. Experimental results on real-world texts and images demonstrate the effectiveness, sample quality as well as efficiency of our designed framework.
  arXiv  Detail & Related papers  (2021-01-18T08:37:13Z)
• Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling [19.24454872492008]
  Weak supervision offers a promising alternative for producing labeled datasets without ground truth labels. We develop the first framework for interactive weak supervision in which a method proposes iterations and learns from user feedback. Our experiments demonstrate that only a small number of feedback are needed to train models that achieve highly competitive test set performance.
  arXiv  Detail & Related papers  (2020-12-11T00:10:38Z)
• An Interpretable and Uncertainty Aware Multi-Task Framework for Multi-Aspect Sentiment Analysis [15.755185152760083]
  Document-level Multi-aspect Sentiment Classification (DMSC) is a challenging and imminent problem. We propose a deliberate self-attention-based deep neural network model, namely FEDAR, for the DMSC problem. FEDAR can achieve competitive performance while also being able to interpret the predictions made.
  arXiv  Detail & Related papers  (2020-09-18T22:32:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.