Towards Requirements Engineering for RAG Systems

• URL: http://arxiv.org/abs/2505.07553v1
• Date: Mon, 12 May 2025 13:30:44 GMT
• Title: Towards Requirements Engineering for RAG Systems
• Authors: Tor Sporsem, Rasmus Ulfsnes,
• Abstract summary: This short paper explores how a maritime company develops and integrates large-language models (LLM)<n>We demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific "retrieval requirements" through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these "retrieval requirements" and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.

Related papers

• Leveraging LLMs for Formal Software Requirements -- Challenges and Prospects [0.0]
  VERIFAI1 aims to investigate automated and semi-automated approaches to bridge this gap.<n>This position paper presents a preliminary synthesis of relevant literature to identify recurring challenges and prospective research directions.
  arXiv  Detail & Related papers  (2025-07-18T19:15:50Z)
• Deep Research Agents: A Systematic Examination And Roadmap [79.04813794804377]
  Deep Research (DR) agents are designed to tackle complex, multi-turn informational research tasks.<n>In this paper, we conduct a detailed analysis of the foundational technologies and architectural components that constitute DR agents.
  arXiv  Detail & Related papers  (2025-06-22T16:52:48Z)
• Hybrid Reasoning for Perception, Explanation, and Autonomous Action in Manufacturing [0.0]
  CIPHER is a vision-language-action (VLA) model framework aiming to replicate human-like reasoning for industrial control.<n>It integrates a process expert, a regression model enabling quantitative characterization of system states.<n>It interprets visual or textual inputs from process monitoring, explains its decisions, and autonomously generates precise machine instructions.
  arXiv  Detail & Related papers  (2025-06-10T05:37:33Z)
• InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation [63.55258191625131]
  InfoDeepSeek is a new benchmark for assessing agentic information seeking in real-world, dynamic web environments.<n>We propose a systematic methodology for constructing challenging queries satisfying the criteria of determinacy, difficulty, and diversity.<n>We develop the first evaluation framework tailored to dynamic agentic information seeking, including fine-grained metrics about the accuracy, utility, and compactness of information seeking outcomes.
  arXiv  Detail & Related papers  (2025-05-21T14:44:40Z)
• Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey [59.52058740470727]
  Edge-cloud collaborative computing (ECCC) has emerged as a pivotal paradigm for addressing the computational demands of modern intelligent applications.<n>Recent advancements in AI, particularly deep learning and large language models (LLMs), have dramatically enhanced the capabilities of these distributed systems.<n>This survey provides a structured tutorial on fundamental architectures, enabling technologies, and emerging applications.
  arXiv  Detail & Related papers  (2025-05-03T13:55:38Z)
• Semi-Automated Design of Data-Intensive Architectures [49.1574468325115]
  This paper introduces a development methodology for data-intensive architectures.<n>It guides architects in (i) designing a suitable architecture for their specific application scenario, and (ii) selecting an appropriate set of concrete systems to implement the application.<n>We show that the description languages we adopt can capture the key aspects of data-intensive architectures proposed by researchers and practitioners.
  arXiv  Detail & Related papers  (2025-03-21T16:01:11Z)
• Causal Models in Requirement Specifications for Machine Learning: A vision [4.348086726793516]
  This vision paper explores causal modelling as an requirements engineering (RE) activity.<n>We propose a workflow to elicit low-level model and data requirements from high-level prior knowledge.<n>The approach is demonstrated on an industrial fault detection system.
  arXiv  Detail & Related papers  (2025-02-17T10:20:17Z)
• PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generation [16.081923602156337]
  We introduce sPecIalized KnowledgE and Rationale Augmentation Generation (PIKE-RAG)<n>We focus on extracting, understanding, and applying specialized knowledge, while constructing coherent rationale to incrementally steer LLMs toward accurate responses.<n>This strategic approach offers a roadmap for the phased development and enhancement of RAG systems, tailored to meet the evolving demands of industrial applications.
  arXiv  Detail & Related papers  (2025-01-20T15:39:39Z)
• Requirements Engineering for a Web-based Research, Technology & Innovation Monitoring Tool [46.38386372048799]
  We introduce a requirements engineering process to identify stakeholders and elicitate requirements for a web-based interactive and open-access RTI system monitoring tool.<n>Based on several core modules, we introduce a multi-tier software architecture of how such a tool is generally implemented from the perspective of software engineers.<n>A cornerstone of this architecture is the user-facing dashboard module.
  arXiv  Detail & Related papers  (2025-01-18T20:36:26Z)
• Accelerating Manufacturing Scale-Up from Material Discovery Using Agentic Web Navigation and Retrieval-Augmented AI for Process Engineering Schematics Design [2.368662284133926]
  Process Flow Diagrams (PFDs) and Process and Instrumentation Diagrams (PIDs) are critical tools for industrial process design, control, and safety.<n>The generation of precise and regulation-compliant diagrams remains a significant challenge, particularly in scaling breakthroughs from material discovery to industrial production in an era of automation and digitalization.<n>This paper introduces an autonomous agentic framework to address these challenges through a twostage approach involving knowledge acquisition and generation.
  arXiv  Detail & Related papers  (2024-12-08T13:36:42Z)
• Towards Generating Executable Metamorphic Relations Using Large Language Models [46.26208489175692]
  We propose an approach for automatically deriving executable MRs from requirements using large language models (LLMs) To assess the feasibility of our approach, we conducted a questionnaire-based survey in collaboration with Siemens Industry Software.
  arXiv  Detail & Related papers  (2024-01-30T13:52:47Z)
• Integration of Domain Expert-Centric Ontology Design into the CRISP-DM for Cyber-Physical Production Systems [45.05372822216111]
  Methods from Machine Learning (ML) and Data Mining (DM) have proven to be promising in extracting complex and hidden patterns from the data collected. However, such data-driven projects, usually performed with the Cross-Industry Standard Process for Data Mining (CRISPDM), often fail due to the disproportionate amount of time needed for understanding and preparing the data. This contribution intends present an integrated approach so that data scientists are able to more quickly and reliably gain insights into the CPPS challenges.
  arXiv  Detail & Related papers  (2023-07-21T15:04:00Z)
• Automated Machine Learning: A Case Study on Non-Intrusive Appliance Load Monitoring [81.06807079998117]
  We propose a novel approach to enable Automated Machine Learning (AutoML) for Non-Intrusive Appliance Load Monitoring (NIALM)<n>NIALM offers a cost-effective alternative to smart meters for measuring the energy consumption of electric devices and appliances.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)
• Technology Readiness Levels for AI & ML [79.22051549519989]
  Development of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. Engineering systems follow well-defined processes and testing standards to streamline development for high-quality, reliable results. We propose a proven systems engineering approach for machine learning development and deployment.
  arXiv  Detail & Related papers  (2020-06-21T17:14:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.