Interview-Informed Generative Agents for Product Discovery: A Validation Study

• URL: http://arxiv.org/abs/2603.29890v1
• Date: Tue, 10 Mar 2026 22:54:45 GMT
• Title: Interview-Informed Generative Agents for Product Discovery: A Validation Study
• Abstract summary: We investigate whether interview-informed generative agents can simulate user responses in concept testing scenarios.<n>Our results show that agents are distribution-calibrated but identity-imprecise.
• Score: 4.775774786716911
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) have shown strong performance on standardized social science instruments, but their value for product discovery remains unclear. We investigate whether interview-informed generative agents can simulate user responses in concept testing scenarios. Using in-depth workflow interviews with knowledge workers, we created personalized agents and compared their evaluations of novel AI concepts against the same participants' responses. Our results show that agents are distribution-calibrated but identity-imprecise: they fail to replicate the specific individual they are grounded in, yet approximate population-level response distributions. These findings highlight both the potential and the limits of LLM simulation in design research. While unsuitable as a substitute for individual-level insights, simulation may provide value for early-stage concept screening and iteration, where distributional accuracy suffices. We discuss implications for integrating simulation responsibly into product development workflows.

Related papers

• Rethinking the Evaluation and Optimization of LLM-Based Social Simulation [56.18577997572273]
  We use the subjectivity coefficient to analyze how accuracy-based evaluation and hard-label training fail as subjectivity grows.<n>We construct SUBJSIM, a benchmark of 19,300 contexts covering 193 annotators and 100 subjective questions.<n>Results on SUBJSIM demonstrate the advantages of our method.
  arXiv  Detail & Related papers  (2026-08-20T06:26:03Z)
• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data [54.145424717168794]
  Large Language Models (LLMs) have demonstrated remarkable human-like capabilities, yet their ability to replicate a specific individual remains under-explored.<n>This paper presents a case study to investigate LLM-based individual simulation with a volunteer-contributed archive of private messaging history spanning over ten years.<n>We propose the "Individual Turing Test" to evaluate whether acquaintances of the volunteer can correctly identify which response in a multi-candidate pool most plausibly comes from the volunteer.
  arXiv  Detail & Related papers  (2026-03-01T21:46:27Z)
• Position: AI Agents Are Not (Yet) a Panacea for Social Simulation [62.891898598784415]
  Recent advances in large language models (LLMs) have spurred growing interest in using LLM-integrated agents for social simulation.<n>This position paper argues that LLM-based agents are not (yet) a panacea for social simulation.
  arXiv  Detail & Related papers  (2026-02-19T06:35:07Z)
• HumanStudy-Bench: Towards AI Agent Design for Participant Simulation [11.906370453952265]
  Large language models (LLMs) are increasingly used as simulated participants in social science experiments.<n>We introduce HUMANSTUDY-BENCH, a benchmark and execution engine that orchestrates LLM-based agents to reconstruct human-subject experiments.<n>To evaluate fidelity at the level of scientific inference, we propose new metrics to quantify how much human and agent behaviors agree.
  arXiv  Detail & Related papers  (2026-01-31T12:07:42Z)
• Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulation [18.70850695450292]
  Large Language Models (LLMs) have fueled the notion that synthetic agents can serve as substitutes for real participants in human-subject research.<n>Do agents maintain internal consistency, retaining similar behaviors when examined under different experimental settings?<n>This study explores whether an agent's conversation behavior is consistent with what we would expect from their revealed internal state.
  arXiv  Detail & Related papers  (2025-09-03T21:55:29Z)
• Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators [13.3155239959814]
  We present a framework for virtual respondent simulation using large language models (LLMs)<n>Our central idea is to account for mediators: factors through which the same trait can give rise to varying responses to a survey item.<n> Experiments on three psychological trait theories (Big5, Schwartz, VIA) show that our mediator generation methods and simulation framework effectively identify high-validity items.
  arXiv  Detail & Related papers  (2025-07-08T11:26:03Z)
• Should you use LLMs to simulate opinions? Quality checks for early-stage deliberation [5.4260854377259315]
  We introduce a quality control assessment designed to test the viability of LLM-simulated opinions on Likert-scale tasks.<n>This assessment comprises two key tests: emphlogical consistency and emphalignment with stakeholder expectations.<n>None of the models or methods pass the full assessment, revealing several failure modes.
  arXiv  Detail & Related papers  (2025-04-11T20:16:02Z)
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations [49.908708778200115]
  We are the first to specialize large language models (LLMs) for simulating survey response distributions.<n>As a testbed, we use country-level results from two global cultural surveys.<n>We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions.
  arXiv  Detail & Related papers  (2025-02-10T21:59:27Z)
• Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation [73.58618024960968]
  An increasing number of studies are employing large language models (LLMs) as agents to emulate the sequential decision-making processes of humans.<n>This arouses curiosity regarding the capacity of LLM agents to comprehend probability distributions.<n>Our analysis indicates that LLM agents can understand probabilities, but they struggle with probability sampling.
  arXiv  Detail & Related papers  (2024-04-13T16:59:28Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.