Reverse Engineering User Stories from Code using Large Language Models

• URL: http://arxiv.org/abs/2509.19587v1
• Date: Tue, 23 Sep 2025 21:23:37 GMT
• Title: Reverse Engineering User Stories from Code using Large Language Models
• Abstract summary: We investigate whether large language models (LLMs) can automatically recover user stories directly from source code.<n>We evaluate five state-of-the-art LLMs across six prompting strategies.<n>Results show that all models achieve, on average, an F1 score of 0.8 for code up to 200 NLOC.
• Score: 19.450989646157716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: User stories are essential in agile development, yet often missing or outdated in legacy and poorly documented systems. We investigate whether large language models (LLMs) can automatically recover user stories directly from source code and how prompt design impacts output quality. Using 1,750 annotated C++ snippets of varying complexity, we evaluate five state-of-the-art LLMs across six prompting strategies. Results show that all models achieve, on average, an F1 score of 0.8 for code up to 200 NLOC. Our findings show that a single illustrative example enables the smallest model (8B) to match the performance of a much larger 70B model. In contrast, structured reasoning via Chain-of-Thought offers only marginal gains, primarily for larger models.

Related papers

• Automated Analysis of Learning Outcomes and Exam Questions Based on Bloom's Taxonomy [0.0]
  This paper explores the automatic classification of exam questions and learning outcomes according to Bloom's taxonomy.<n>A small dataset of 600 sentences labeled with six cognitive categories was processed using traditional machine learning (ML) models.
  arXiv  Detail & Related papers  (2025-11-14T02:31:12Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension [77.93156509994994]
  We show how to represent short chunks in a way that is conditioned on a broader context window to enhance retrieval performance.<n>Existing embedding models are not well-equipped to encode such situated context effectively.<n>Our method substantially outperforms state-of-the-art embedding models.
  arXiv  Detail & Related papers  (2025-08-03T23:59:31Z)
• Token Level Routing Inference System for Edge Devices [21.721914273034972]
  We present a novel collaborative decoding inference system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation.<n>Remarkably, the system achieves a 60% performance gain on CommonsenseQA using only a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud.
  arXiv  Detail & Related papers  (2025-04-10T15:54:19Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models [64.18350535770357]
  We propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning.<n>Our approach only leverages a small number of samples to search for the desired pruning policy.<n>We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering.
  arXiv  Detail & Related papers  (2025-03-19T16:07:04Z)
• Model Hubs and Beyond: Analyzing Model Popularity, Performance, and Documentation [1.2888930658406668]
  We evaluate a comprehensive set of 500 Sentiment Analysis models on Hugging Face.<n>Our findings reveal that model popularity does not necessarily correlate with performance.<n>About 88% of model authors overstate their models' performance in the model cards.
  arXiv  Detail & Related papers  (2025-03-19T14:01:33Z)
• Why Personalizing Deep Learning-Based Code Completion Tools Matters [55.39571645315926]
  We consider 136 developers from two organizations (Apache and Spring), two model architectures (T5 and Code Llama), and three model sizes (60M, 750M, and 7B trainable parameters)<n>For the Code Llama model (7B), we compared the performance of the already pre-trained model publicly available online with the same model fine-tuned on organization- and developer-specific datasets.<n>Our results show that there is a boost in prediction capabilities provided by both an organization-specific and a developer-specific additional fine-tuning.
  arXiv  Detail & Related papers  (2025-03-18T12:26:06Z)
• Large Language Model-guided Document Selection [23.673690115025913]
  Large Language Model (LLM) pre-training exhausts an ever growing compute budget. Recent research has demonstrated that careful document selection enables comparable model quality with only a fraction of the FLOPs. We explore a promising direction for scalable general-domain document selection.
  arXiv  Detail & Related papers  (2024-06-07T04:52:46Z)
• Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification [4.4467858321751015]
  We benchmark language models from 77M to 40B parameters using different architectures and scoring functions. Our findings reveal that small models can effectively classify texts, getting on par with or surpassing their larger counterparts. This research underscores the notion that bigger isn't always better, suggesting that resource-efficient small models may offer viable solutions for specific data classification challenges.
  arXiv  Detail & Related papers  (2024-04-17T07:10:28Z)
• Identifying and Mitigating Model Failures through Few-shot CLIP-aided Diffusion Generation [65.268245109828]
  We propose an end-to-end framework to generate text descriptions of failure modes associated with spurious correlations. These descriptions can be used to generate synthetic data using generative models, such as diffusion models. Our experiments have shown remarkable textbfimprovements in accuracy ($sim textbf21%$) on hard sub-populations.
  arXiv  Detail & Related papers  (2023-12-09T04:43:49Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [49.15931834209624]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world.<n>We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.<n>By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.