From Reviews to Requirements: Can LLMs Generate Human-Like User Stories?

• URL: http://arxiv.org/abs/2603.28163v1
• Date: Mon, 30 Mar 2026 08:31:18 GMT
• Title: From Reviews to Requirements: Can LLMs Generate Human-Like User Stories?
• Abstract summary: We evaluate how well large language models (LLMs) can generate usable user stories directly from raw app reviews.<n>Using the Mini-BAR dataset of 1,000+ health app reviews, we tested zero-shot, one-shot, and two-shot prompting methods.<n>Our results show that LLMs can match or even outperform humans in writing fluent, well-formatted user stories.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: App store reviews provide a constant flow of real user feedback that can help improve software requirements. However, these reviews are often messy, informal, and difficult to analyze manually at scale. Although automated techniques exist, many do not perform well when replicated and often fail to produce clean, backlog-ready user stories for agile projects. In this study, we evaluate how well large language models (LLMs) such as GPT-3.5 Turbo, Gemini 2.0 Flash, and Mistral 7B Instruct can generate usable user stories directly from raw app reviews. Using the Mini-BAR dataset of 1,000+ health app reviews, we tested zero-shot, one-shot, and two-shot prompting methods. We evaluated the generated user stories using both human judgment (via the RUST framework) and a RoBERTa classifier fine-tuned on UStAI to assess their overall quality. Our results show that LLMs can match or even outperform humans in writing fluent, well-formatted user stories, especially when few-shot prompts are used. However, they still struggle to produce independent and unique user stories, which are essential for building a strong agile backlog. Overall, our findings show how LLMs can reliably turn unstructured app reviews into actionable software requirements, providing developers with clear guidance to turn user feedback into meaningful improvements.

Related papers

• From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs [48.36707944399574]
  We study how vibe-testing works in practice and then formalize it to support systematic analysis.<n>In experiments on coding benchmarks, we find that combining personalized prompts and user-aware evaluation can change which model is preferred.
  arXiv  Detail & Related papers  (2026-04-15T17:57:08Z)
• Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users [48.61084675902644]
  Deep Research (DR) tools can synthesize scientific papers to answer researchers' queries, but lack understanding of their users.<n>We develop a personalized DR tool that infers a profile of a user's research interests and proposes personalized actions for a user's input query.<n>We reveal nine nuanced errors of personalized DR undetectable by our LLM judges, and we study qualitative feedback to form lessons for future DR design.
  arXiv  Detail & Related papers  (2026-03-17T04:59:32Z)
• Few-Shot and Training-Free Review Generation via Conversational Prompting [2.0305676256390934]
  Real-world applications often face few-shot and training-free situations.<n>We propose Conversational Prompting, a lightweight method that reformulates user reviews as multi-turn conversations.
  arXiv  Detail & Related papers  (2025-09-25T06:36:08Z)
• User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal [59.120335322495436]
  We analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs.<n>Second, we study harvesting learning signals from such implicit user feedback.
  arXiv  Detail & Related papers  (2025-07-30T23:33:29Z)
• Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback [57.200668979963694]
  We present a novel test set of 1,300 stories that we corrupted to intentionally introduce writing issues.<n>We study the performance of commonly used LLMs in this task with both automatic and human evaluation metrics.
  arXiv  Detail & Related papers  (2025-07-21T18:56:50Z)
• Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale [53.059480071818136]
  Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks.<n> PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories.<n>We evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's profile.
  arXiv  Detail & Related papers  (2025-04-19T08:16:10Z)
• Exploring LLMs Impact on Student-Created User Stories and Acceptance Testing in Software Development [0.0]
  This study investigates how LLMs (large language models) affect undergraduate software engineering students' ability to transform user feedback into user stories.<n>Students, working individually, were asked to analyze user feedback comments, appropriately group related items, and create user stories.<n>We found that LLMs help students develop valuable stories with well-defined acceptance criteria.
  arXiv  Detail & Related papers  (2025-02-04T19:35:44Z)
• User Story Tutor (UST) to Support Agile Software Developers [0.4077787659104315]
  We designed, implemented, applied, and evaluated a web application called User Story Tutor (UST) UST checks the description of a given User Story for readability, and if needed, recommends appropriate practices for improvement. UST may support the continuing education of agile development teams when writing and reviewing User Stories.
  arXiv  Detail & Related papers  (2024-06-24T01:55:01Z)
• Zero-shot Bilingual App Reviews Mining with Large Language Models [0.7340017786387767]
  Mini-BAR is a tool that integrates large language models (LLMs) to perform zero-shot mining of user reviews in both English and French. To evaluate the performance of Mini-BAR, we created a dataset containing 6,000 English and 6,000 French annotated user reviews.
  arXiv  Detail & Related papers  (2023-11-06T12:36:46Z)
• Recommendations by Concise User Profiles from Review Text [24.408292545170944]
  This work addresses the difficult and underexplored case of users who have very sparse interactions but post informative review texts.<n> feeding the full text of all reviews through an LLM has a weak signal-to-noise ratio and incurs high costs of processed tokens.<n>It presents a light-weight framework, called CUP, which first computes concise user profiles and feeds only these into the training of transformer-based recommenders.
  arXiv  Detail & Related papers  (2023-11-02T15:31:12Z)
• Automating App Review Response Generation [67.58267006314415]
  We propose a novel approach RRGen that automatically generates review responses by learning knowledge relations between reviews and their responses. Experiments on 58 apps and 309,246 review-response pairs highlight that RRGen outperforms the baselines by at least 67.4% in terms of BLEU-4.
  arXiv  Detail & Related papers  (2020-02-10T05:23:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.