Towards Automated Safety Requirements Derivation Using Agent-based RAG

• URL: http://arxiv.org/abs/2504.11243v1
• Date: Tue, 15 Apr 2025 14:43:19 GMT
• Title: Towards Automated Safety Requirements Derivation Using Agent-based RAG
• Abstract summary: We study the automated derivation of safety requirements in a self-driving vehicle use case, leveraging LLMs in combination with agent-based retrieval-augmented generation.<n>Existing RAG approaches address this issue, yet their performance deteriorates when handling complex queries.<n>In this paper, we propose the use of agent-based RAG to derive safety requirements and show that the retrieved information is more relevant to the queries.
• Score: 0.25128687379089687
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the automated derivation of safety requirements in a self-driving vehicle use case, leveraging LLMs in combination with agent-based retrieval-augmented generation. Conventional approaches that utilise pre-trained LLMs to assist in safety analyses typically lack domain-specific knowledge. Existing RAG approaches address this issue, yet their performance deteriorates when handling complex queries and it becomes increasingly harder to retrieve the most relevant information. This is particularly relevant for safety-relevant applications. In this paper, we propose the use of agent-based RAG to derive safety requirements and show that the retrieved information is more relevant to the queries. We implement an agent-based approach on a document pool of automotive standards and the Apollo case study, as a representative example of an automated driving perception system. Our solution is tested on a data set of safety requirement questions and answers, extracted from the Apollo data. Evaluating a set of selected RAG metrics, we present and discuss advantages of a agent-based approach compared to default RAG methods.

Related papers

• Breadcrumbing Search Agents [118.20284823674382]
  LLM-based search agents are widely used for information-seeking tasks, but their reliance on external tool returns introduces a critical security risk.<n>We show that the channel delivering search and page observations is a fragile security boundary.<n>Under a constrained tool-intermediary threat model, appending only one controlled result per query can substantially increase attack success.
  arXiv  Detail & Related papers  (2026-08-05T07:57:27Z)
• Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learning [42.850404278428954]
  We introduce robust dynamic few-shot learning which retrieves examples from relevant user queries.<n>We show that our system can be self-learned, for instance on prior user queries, and is therefore easily applicable in practice.
  arXiv  Detail & Related papers  (2026-01-13T00:56:38Z)
• Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
  Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
  arXiv  Detail & Related papers  (2026-01-12T21:31:38Z)
• Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval [49.85856484781787]
  We introduce Interact-RAG, a new paradigm that elevates the LLM agent into an active manipulator of the retrieval process.<n>We develop a reasoning-enhanced workflow, which enables both zero-shot execution and the synthesis of interaction trajectories.<n>Experiments across six benchmarks demonstrate that Interact-RAG significantly outperforms other advanced methods.
  arXiv  Detail & Related papers  (2025-10-31T15:48:43Z)
• ATAG: AI-Agent Application Threat Assessment with Attack Graphs [23.757154032523093]
  This paper introduces AI-agent application Threat assessment with Attack Graphs (ATAG)<n>ATAG is a novel framework designed to systematically analyze the security risks associated with AI-agent applications.<n>It facilitates proactive identification and mitigation of AI-agent threats in multi-agent applications.
  arXiv  Detail & Related papers  (2025-06-03T13:25:40Z)
• SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulator [77.86600052899156]
  Large Language Model (LLM)-based agents are increasingly deployed in real-world applications.<n>We propose AutoSafe, the first framework that systematically enhances agent safety through fully automated synthetic data generation.<n>We show that AutoSafe boosts safety scores by 45% on average and achieves a 28.91% improvement on real-world tasks.
  arXiv  Detail & Related papers  (2025-05-23T10:56:06Z)
• Securing RAG: A Risk Assessment and Mitigation Framework [0.0]
  Retrieval Augmented Generation (RAG) has emerged as the de facto industry standard for user-facing NLP applications.<n>This paper reviews the vulnerabilities of RAG pipelines, and outlines the attack surface from data pre-processing to integration with Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2025-05-13T16:39:00Z)
• SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model [17.046058202577985]
  We introduce a benchmark named SafeRAG designed to evaluate the RAG security.<n>First, we classify attack tasks into silver noise, inter-context conflict, soft ad, and white Denial-of-Service.<n>We then utilize the SafeRAG dataset to simulate various attack scenarios that RAG may encounter.
  arXiv  Detail & Related papers  (2025-01-28T17:01:31Z)
• Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG [0.8463972278020965]
  Large Language Models (LLMs) have revolutionized artificial intelligence (AI) by enabling human like text generation and natural language understanding.<n>Retrieval Augmented Generation (RAG) has emerged as a solution, enhancing LLMs by integrating real time data retrieval to provide contextually relevant responses.<n>Agentic Retrieval-Augmented Generation (RAG) transcends these limitations by embedding autonomous AI agents into the RAG pipeline.
  arXiv  Detail & Related papers  (2025-01-15T20:40:25Z)
• Unanswerability Evaluation for Retrieval Augmented Generation [74.3022365715597]
  UAEval4RAG is a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively. We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries.
  arXiv  Detail & Related papers  (2024-12-16T19:11:55Z)
• Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversations [0.0]
  In customer contact centers, human agents often struggle with long average handling times (AHT) We propose a decision support system that can look beyond RAG by first identifying customer questions in real time. If the query matches an FAQ, the system retrieves the answer directly from the FAQ database; otherwise, it generates answers via RAG.
  arXiv  Detail & Related papers  (2024-10-14T04:06:22Z)
• DeepNote: Note-Centric Deep Retrieval-Augmented Generation [72.70046559930555]
  Retrieval-Augmented Generation (RAG) mitigates factual errors and hallucinations in Large Language Models (LLMs) for question-answering (QA)<n>We develop DeepNote, an adaptive RAG framework that achieves in-depth and robust exploration of knowledge sources through note-centric adaptive retrieval.
  arXiv  Detail & Related papers  (2024-10-11T14:03:29Z)
• Enhancing Retrieval in QA Systems with Derived Feature Association [0.0]
  Retrieval augmented generation (RAG) has become the standard in long context question answering (QA) systems. We propose a novel extension to RAG systems, which we call Retrieval from AI Derived Documents (RAIDD)
  arXiv  Detail & Related papers  (2024-10-02T05:24:49Z)
• RAG based Question-Answering for Contextual Response Prediction System [0.4660328753262075]
  Large Language Models (LLMs) have shown versatility in various Natural Language Processing (NLP) tasks. Retrieval Augmented Generation (RAG) emerges as a promising technique to address this challenge. This paper introduces an end-to-end framework that employs LLMs with RAG capabilities for industry use cases.
  arXiv  Detail & Related papers  (2024-09-05T17:14:23Z)
• "Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models [74.05368440735468]
  Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases.
  arXiv  Detail & Related papers  (2024-06-26T05:36:23Z)
• REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering [115.72130322143275]
  REAR is a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA) We develop a novel architecture for LLM-based RAG systems, by incorporating a specially designed assessment module. Experiments on four open-domain QA tasks show that REAR significantly outperforms previous a number of competitive RAG approaches.
  arXiv  Detail & Related papers  (2024-02-27T13:22:51Z)
• On Prompt-Driven Safeguarding for Large Language Models [172.13943777203377]
  We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO. Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness.
  arXiv  Detail & Related papers  (2024-01-31T17:28:24Z)
• A Counterfactual Safety Margin Perspective on the Scoring of Autonomous Vehicles' Riskiness [52.27309191283943]
  This paper presents a data-driven framework for assessing the risk of different AVs' behaviors. We propose the notion of counterfactual safety margin, which represents the minimum deviation from nominal behavior that could cause a collision.
  arXiv  Detail & Related papers  (2023-08-02T09:48:08Z)
• Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation [79.89605349842569]
  We introduce the Collection and Refinement of Online Properties (CROP) framework to design properties at training time. CROP employs a cost signal to identify unsafe interactions and use them to shape safety properties. We evaluate our approach in several robotic mapless navigation tasks and demonstrate that the violation metric computed with CROP allows higher returns and lower violations over previous Safe DRL approaches.
  arXiv  Detail & Related papers  (2023-02-13T21:19:36Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.