Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations

• URL: http://arxiv.org/abs/2604.22207v1
• Date: Fri, 24 Apr 2026 04:22:17 GMT
• Title: Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations
• Abstract summary: We discuss a possible approach for automating the Goal-Oriented Requirements Engineering (GORE) process by extracting functional goals from software documentation.<n>To implement these functionalities, we propose a chain of Large Language Models fed with engineered prompts.<n>Although the pipeline achieved 61% accuracy in low-level goal identification, the final stage, these results indicate the approach is best suited as a tool to accelerate manual extraction.
• Score: 4.451267761568192
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Due to the textual and repetitive nature of many Requirements Engineering (RE) artefacts, Large Language Models (LLMs) have proven useful to automate their generation and processing. In this paper, we discuss a possible approach for automating the Goal-Oriented Requirements Engineering (GORE) process by extracting functional goals from software documentation through three phases: actor identification, high and low-level goal extraction. To implement these functionalities, we propose a chain of LLMs fed with engineered prompts. We experimented with different variants of in-context learning and measured the similarities between input data and in-context examples to better investigate their impact. Another key element is the generation-critic mechanism, implemented as a feedback loop involving two LLMs. Although the pipeline achieved 61% accuracy in low-level goal identification, the final stage, these results indicate the approach is best suited as a tool to accelerate manual extraction rather than as a full replacement. The feedback-loop mechanism with Zero-shot outperformed stand-alone Few-shot, with an ablation study suggesting that performance slightly degrades without the feedback cycle. However, we reported that the combination of the feedback mechanism with Few-shot does not deliver any advantage, possibly suggesting that the primary performance ceiling is the prompting strategy applied to the 'critic' LLM. Together with the refinement of both the quantity and quality of the Shot examples, future research will integrate Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting to improve accuracy.

Related papers

• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis [78.32151470154422]
  We introduce RAVEL, an agentic framework that enables the testers to autonomously plan and execute typical synthesis operations.<n>We present C3EBench, a benchmark comprising 1,258 samples derived from professional human writings.<n>By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability.
  arXiv  Detail & Related papers  (2026-02-28T14:47:34Z)
• From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs [48.83701310501069]
  Large language models (LLMs) have achieved notable performance in code synthesis.<n>We introduce a performance-aware, closed-loop solution that enables LLMs to autonomously engineer optimal transformations.<n>We fine-tune LLMs with Low-Rank Adaptation on a novel repository of more than 6,000 empirically evaluated PyTorch augmentation functions.
  arXiv  Detail & Related papers  (2026-01-07T11:13:02Z)
• SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models [10.04930078540686]
  SCRIBE is a reinforcement learning framework that intervenes at a novel mid-level abstraction.<n>It achieves state-of-the-art performance across a range of reasoning and tool-use benchmarks.
  arXiv  Detail & Related papers  (2026-01-07T03:49:48Z)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding [70.01564145836129]
  This paper conceptualizes the decoding process as analogous to the ranking stage in recommendation pipelines.<n>Motivated by this insight, we propose Language Ranker, a novel framework that introduces a lightweight module to rerank candidate responses.<n> Experiments show that Language Ranker achieves performance comparable to large-scale reward models, while requiring only 0.5M additional parameters.
  arXiv  Detail & Related papers  (2025-10-23T17:56:46Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• Your Coding Intent is Secretly in the Context and You Should Deliberately Infer It Before Completion [18.177415287934487]
  Large Language Models (LLMs) are increasingly used for function completion in repository-scales.<n>In real-world repositories, such annotations are frequently absent, and performance drops substantially without them.<n>We frame the task as a three-stage process. The first stage focuses on intent inference, where the model analyzes the code preceding the target function to uncover cues about the desired functionality.<n>The second stage introduces an optional interactive refinement mechanism to handle cases where preceding context alone is insufficient for intent recovery.
  arXiv  Detail & Related papers  (2025-08-13T06:45:23Z)
• LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts [2.685668802278156]
  We introduce LLMQuoter, a lightweight, distillation-based model designed to enhance Retrieval Augmented Generation (RAG)<n>Built on the LLaMA-3B architecture and fine-tuned with Low-Rank Adaptation (LoRA) on a 15,000-sample subset of HotpotQA, LLMQuoter adopts a "quote-first-then-answer" strategy, efficiently identifying key quotes before passing curated snippets to reasoning models.<n>This workflow reduces cognitive overhead and outperforms full-context approaches like Retrieval-Augmented Fine-Tuning (RAFT), achieving over 20-point accuracy gains across both small and large language
  arXiv  Detail & Related papers  (2025-01-09T20:01:15Z)
• Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting [21.04933334040135]
  We introduce the Self-Prompting framework, a novel method designed to fully harness the embedded RE knowledge within Large Language Models.<n>Our framework employs a three-stage diversity approach to prompt LLMs, generating multiple synthetic samples that encapsulate specific relations from scratch.<n> Experimental evaluations on benchmark datasets show our approach outperforms existing LLM-based zero-shot RE methods.
  arXiv  Detail & Related papers  (2024-10-02T01:12:54Z)
• LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback [65.84061725174269]
  Recent large language models (LLM) are leveraging human feedback to improve their generation quality. We propose LLMRefine, an inference time optimization method to refine LLM's output. We conduct experiments on three text generation tasks, including machine translation, long-form question answering (QA), and topical summarization. LLMRefine consistently outperforms all baseline approaches, achieving improvements up to 1.7 MetricX points on translation tasks, 8.1 ROUGE-L on ASQA, 2.2 ROUGE-L on topical summarization.
  arXiv  Detail & Related papers  (2023-11-15T19:52:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.