An Industrial-Scale Retrieval-Augmented Generation Framework for Requirements Engineering: Empirical Evaluation with Automotive Manufacturing Data

• URL: http://arxiv.org/abs/2603.20534v2
• Date: Tue, 24 Mar 2026 17:44:26 GMT
• Title: An Industrial-Scale Retrieval-Augmented Generation Framework for Requirements Engineering: Empirical Evaluation with Automotive Manufacturing Data
• Abstract summary: Retrieval-augmented generation (RAG) shows promise for knowledge-intensive tasks, but no prior work has evaluated RAG on authentic industrial RE.<n>This paper presents a comprehensive empirical evaluation of RAG for industrial requirements engineering automation using authentic automotive manufacturing documentation.<n>RAG achieves 98.2% extraction accuracy with complete traceability, outperforming baselines by 24.4% and 19.6%, respectively.
• Score: 0.5156484100374058
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Requirements engineering in Industry 4.0 faces critical challenges with heterogeneous, unstructured documentation spanning technical specifications, supplier lists, and compliance standards. While retrieval-augmented generation (RAG) shows promise for knowledge-intensive tasks, no prior work has evaluated RAG on authentic industrial RE workflows using comprehensive production-grade performance metrics. This paper presents a comprehensive empirical evaluation of RAG for industrial requirements engineering automation using authentic automotive manufacturing documentation comprising 669 requirements across four specification standards (MBN 9666-1, MBN 9666-2, BQF 9666-5, MBN 9666-9) spanning 2015-2023, plus 49 supplier qualifications with extensive supporting documentation. Through controlled comparisons with BERT-based and ungrounded LLM approaches, the framework achieves 98.2% extraction accuracy with complete traceability, outperforming baselines by 24.4% and 19.6%, respectively. Hybrid semantic-lexical retrieval achieves MRR of 0.847. Expert quality assessment averaged 4.32/5.0 across five dimensions. The evaluation demonstrates 83% reduction in manual analysis time and 47% cost savings through multi-provider LLM orchestration. Ablation studies quantify individual component contributions. Longitudinal analysis reveals a 55% reduction in requirement volume coupled with 1,800% increase in IT security focus, identifying 10 legacy suppliers (20.4%) requiring requalification, representing potential $2.3M in avoided contract penalties.

Related papers

• EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval [41.53357936032648]
  Existing benchmarks assume clean retrieval with simple queries, failing to capture production conditions where noisy documents and multi-dimensional constraints coexist.<n>We introduce EnterpriseRAG, a benchmark of 983 expert-validated samples across six domains.<n>We will release the benchmark and evaluation framework upon publication.
  arXiv  Detail & Related papers  (2026-08-12T02:51:04Z)
• MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loop [0.0]
  MADP is a multi-agent architecture that combines deep learning-based classification and parsing with large language model extraction.<n>Production deployment on 955 real-world documents processed through January 2026 achieves a 97.0% full-pipeline automation rate.<n>Our hybrid AI+HITL approach reduces CO2 emissions by 69%, energy consumption by 69%, and water usage by 63% compared to traditional manual processing.
  arXiv  Detail & Related papers  (2026-05-16T21:18:39Z)
• Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMs [37.25886192772028]
  We propose OPT-BENCH, the first comprehensive framework for training and evaluating Large Language Models (LLMs)<n> OPT-BENCH provides three key components: a scalable training infrastructure with instance generators, quality verifiers, and optimal baselines across 10 tasks; a rigorous benchmark with 1,000 instances evaluating both feasibility, measured by Success Rate, and quality, measured by Quality Ratio; and quality-aware rewards that enable continuous improvement beyond binary correctness.<n>Our analysis reveals that quality-aware rewards improve solutions by 28.8% over binary rewards, and that task diversity drives generalization more than data quantity, offering insights into RLVR scaling for complex reasoning
  arXiv  Detail & Related papers  (2026-05-09T11:57:25Z)
• OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation [57.505743202759646]
  OccuBench is a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains.<n>Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity.
  arXiv  Detail & Related papers  (2026-04-13T00:27:32Z)
• Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG [56.95387658211215]
  We introduce Synthetic Mixed Training, which combines synthetic QAs and synthetic documents.<n>This enables log-linear improvements as both synthetic data volume and generator strength increase.<n>Across models and benchmarks, our training enables models to beat RAG in five of six settings, outperforms by 2.6%, and achieves a 9.1% gain when combined with RAG.
  arXiv  Detail & Related papers  (2026-03-24T04:32:48Z)
• CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production [52.85500933801205]
  CharacterFlywheel is an iterative process for improving large language models (LLMs) in production social chat applications.<n>We refined models across 15 generations using data from both internal and external real-user traffic.<n>We conducted 7-day A/B tests showing consistent engagement improvements.
  arXiv  Detail & Related papers  (2026-03-02T15:27:31Z)
• Intelligent Reservoir Decision Support: An Integrated Framework Combining Large Language Models, Advanced Prompt Engineering, and Multimodal Data Fusion for Real-Time Petroleum Operations [0.0]
  This study presents a novel integrated framework combining state-of-the-art large language models with advanced prompt engineering techniques and multimodal data fusion for comprehensive reservoir analysis.<n>The framework implements domain-specific retrieval-augmented generation (RAG) with over 50,000 petroleum engineering documents, chain-of-thought reasoning, and few-shot learning for rapid field adaptation.<n>The system achieves sub-second response times while maintaining 96.2% safety reliability with no high-risk incidents during evaluation.
  arXiv  Detail & Related papers  (2025-09-14T18:13:27Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models [51.55869466207234]
  Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting.<n>We introduce LLMEval-3, a framework for dynamic evaluation of LLMs.<n>LLEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run.
  arXiv  Detail & Related papers  (2025-08-07T14:46:30Z)
• QUEST: Quality-aware Semi-supervised Table Extraction for Business Documents [0.1516287840715525]
  QUEST is a Quality-aware Semi-supervised Table extraction framework for business documents.<n>QUEST evaluates structural and contextual features of extracted tables, trained to predict F1 scores instead of relying on confidence metrics.
  arXiv  Detail & Related papers  (2025-06-17T14:25:44Z)
• ESGenius: Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledge [40.49917730563565]
  ESGenius is a comprehensive benchmark for evaluating and enhancing the proficiency of Large Language Models (LLMs) in Environmental, Social, and Governance (ESG)<n> ESGenius comprises two key components: (i) ESGenius-QA, a collection of 1,136 Multiple-Choice Questions (MCQs) generated by LLMs and rigorously validated by domain experts, covering a broad range of ESG pillars and sustainability topics; and (ii) ESGenius-Corpus, a meticulously curated repository of 231 foundational frameworks, standards, reports, and recommendation documents from 7 authoritative sources.
  arXiv  Detail & Related papers  (2025-06-02T13:19:09Z)
• Automated Non-Functional Requirements Generation in Software Engineering with Large Language Models: A Comparative Study [0.0]
  Non-functional requirements (NFRs) are often overlooked or difficult to identify, impacting software quality.<n>We developed a framework that leverages Large Language Models (LLMs) to derive quality-driven NFRs from functional requirements (FRs)<n>Using a custom prompting technique within a Deno-based pipeline, the system identifies relevant quality attributes for each functional requirement and generates corresponding NFRs.
  arXiv  Detail & Related papers  (2025-03-19T14:23:22Z)
• Classification or Prompting: A Case Study on Legal Requirements Traceability [4.629156733452248]
  Legal requirements traceability is a key task where engineers must analyze technical requirements against target artifacts.<n>In this paper, we investigate two automated solutions based on language models, including large ones (LLMs)<n>The first solution, Kashif, is a classifier that leverages sentence transformers and semantic similarity.<n>The second solution, RICE_LRT, prompts a recent generative LLM based on RICE, a prompt engineering framework.
  arXiv  Detail & Related papers  (2025-02-07T13:33:40Z)
• The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility? [54.18519360412294]
  Large Language Models (LLMs) must balance between rejecting harmful requests for safety and accommodating legitimate ones for utility.<n>This paper presents a Direct Preference Optimization (DPO) based alignment framework that achieves better overall performance.<n>We analyze experimental results obtained from testing DeepSeek-R1 on our benchmark and reveal the critical ethical concerns raised by this highly acclaimed model.
  arXiv  Detail & Related papers  (2025-01-20T06:35:01Z)
• How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts [54.07541591018305]
  We present MAD-Bench, a benchmark that contains 1000 test samples divided into 5 categories, such as non-existent objects, count of objects, and spatial relationship. We provide a comprehensive analysis of popular MLLMs, ranging from GPT-4v, Reka, Gemini-Pro, to open-sourced models, such as LLaVA-NeXT and MiniCPM-Llama3. While GPT-4o achieves 82.82% accuracy on MAD-Bench, the accuracy of any other model in our experiments ranges from 9% to 50%.
  arXiv  Detail & Related papers  (2024-02-20T18:31:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.