Data-Driven Persona-Conditioned Agents for A/B Test Simulation

• URL: http://arxiv.org/abs/2609.01038v1
• Date: Tue, 01 Sep 2026 10:35:49 GMT
• Title: Data-Driven Persona-Conditioned Agents for A/B Test Simulation
• Abstract summary: We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas.<n>Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data-activity patterns.<n>Our best configuration achieves 0.75-0.90 directional accuracy depending on the test metric.
• Score: 14.4702522133194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data-activity patterns, engagement signals, and inferred demographics-enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75-0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.

Related papers

• Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data [73.19814469893088]
  Digital twins and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems.<n>This paper addresses whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible.<n>Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing.
  arXiv  Detail & Related papers  (2026-08-28T07:05:38Z)
• SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation [3.609531017498719]
  We present SimAB, a system that reframes A/B testing as a fast, privacy-preserving simulation using persona-conditioned AI agents.<n>Given design screenshots and a conversion goal, SimAB generates user personas, deploys them as agents that state their preference, aggregates results, and synthesizes rationales.
  arXiv  Detail & Related papers  (2026-03-01T10:08:27Z)
• HumanStudy-Bench: Towards AI Agent Design for Participant Simulation [11.906370453952265]
  Large language models (LLMs) are increasingly used as simulated participants in social science experiments.<n>We introduce HUMANSTUDY-BENCH, a benchmark and execution engine that orchestrates LLM-based agents to reconstruct human-subject experiments.<n>To evaluate fidelity at the level of scientific inference, we propose new metrics to quantify how much human and agent behaviors agree.
  arXiv  Detail & Related papers  (2026-01-31T12:07:42Z)
• Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing [54.456400601801704]
  We introduce a multi-modal user agent for A/B testing (A/B Agent)<n>Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions.<n>We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features.
  arXiv  Detail & Related papers  (2026-01-08T03:33:43Z)
• PUB: An LLM-Enhanced Personality-Driven User Behaviour Simulator for Recommender System Evaluation [9.841963696576546]
  Personality-driven User Behaviour Simulator (PUB) integrates the Big Five personality traits to model personalised user behaviour.<n>PUB dynamically infers user personality from behavioural logs (e.g., ratings, reviews) and item metadata, then generates synthetic interactions that preserve statistical fidelity to real-world data.<n> Experiments on the Amazon review datasets show that logs generated by PUB closely align with real user behaviour and reveal meaningful associations between personality traits and recommendation outcomes.
  arXiv  Detail & Related papers  (2025-06-05T01:57:36Z)
• TestAgent: An Adaptive and Intelligent Expert for Human Assessment [62.060118490577366]
  We propose TestAgent, a large language model (LLM)-powered agent designed to enhance adaptive testing through interactive engagement.<n>TestAgent supports personalized question selection, captures test-takers' responses and anomalies, and provides precise outcomes through dynamic, conversational interactions.
  arXiv  Detail & Related papers  (2025-06-03T16:07:54Z)
• AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents [35.8650712223701]
  A/B testing remains constrained by its dependence on the large-scale and live traffic of human participants.<n>We present AgentA/B, a novel system that automatically simulate user interaction behaviors with real webpages.<n>Our findings suggest AgentA/B can emulate human-like behavior patterns.
  arXiv  Detail & Related papers  (2025-04-13T21:10:56Z)
• Downstream-Pretext Domain Knowledge Traceback for Active Learning [138.02530777915362]
  We propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. Experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-20T01:34:13Z)
• ASPEST: Bridging the Gap Between Active Learning and Selective Prediction [56.001808843574395]
  Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
  arXiv  Detail & Related papers  (2023-04-07T23:51:07Z)
• Delving into Identify-Emphasize Paradigm for Combating Unknown Bias [52.76758938921129]
  We propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy. We also propose gradient alignment (GA) to balance the contributions of the mined bias-aligned and bias-conflicting samples. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases.
  arXiv  Detail & Related papers  (2023-02-22T14:50:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.