Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments

• URL: http://arxiv.org/abs/2505.01307v1
• Date: Fri, 02 May 2025 14:34:33 GMT
• Title: Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments
• Abstract summary: Document Retrieval-Augmented Fine-Tuning (DRAFT) is a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment.<n>DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework.<n>Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model.
• Score: 0.9894420655516565
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents Document Retrieval-Augmented Fine-Tuning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework that accommodates our dual-retrieval architecture, which simultaneously accesses both software documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodology that incorporates variable numbers of relevant documents with meaningful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative improvements in evidence handling, response structure, and domain-specific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transparency and evidence-based reasoning essential in regulatory domains.

Related papers

• The Evaluation Context Protocol (ECP): A Portable Contract for AI Agent Evaluation [0.0]
  This paper explores the critical methodologies for evaluating AI agents and the essential role of advanced observability infrastructure.<n>We propose the Evaluation Context Protocol (ECP), an early-stage, vendor-neutral framework intended to act as a contract layer for agentic systems.<n>We describe an open-source reference implementation that includes adapters for LangChain, LlamaIndex, CrewAI, and PydanticAI.
  arXiv  Detail & Related papers  (2026-08-18T07:14:36Z)
• An Agentic Retrieval Framework for Autonomous Context-Aware Data Quality Assessment [0.0]
  We propose a unified agentic-retrieval framework for autonomous context-aware data quality assessment.<n>The framework interprets natural-language descriptions of intended data usage, derives context-aware assessment strategies, and generates executable validation logic.<n>We implement the proposed framework as an end-to-end prototype and evaluate it across multiple usage scenarios applied to the same dataset.
  arXiv  Detail & Related papers  (2026-05-15T11:24:53Z)
• Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation [50.85337196571056]
  Text-to-image (T2I) generation has advanced rapidly, making reliable evaluation critical as performance differences between models narrow.<n>Existing evaluation practices typically apply uniform annotation mechanisms, such as Likert-scale or binary question answering (BQA)<n>We revisit T2I evaluation through the lens of skill-aligned annotation, where annotation strategies reflect the underlying characteristics of each evaluation skill.
  arXiv  Detail & Related papers  (2026-05-13T09:14:31Z)
• A Sanity Check on Composed Image Retrieval [91.95275287747499]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification.<n>We introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs.<n>We propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios.
  arXiv  Detail & Related papers  (2026-04-14T15:52:22Z)
• Chunking, Retrieval, and Re-ranking: An Empirical Evaluation of RAG Architectures for Policy Document Question Answering [0.0]
  The integration of Large Language Models (LLMs) into the public health policy sector offers a transformative approach to navigating the vast repositories of regulatory guidance maintained by agencies such as the Centers for Disease Control and Prevention (CDC)<n>The propensity for LLMs to generate hallucinations, defined as plausible but factually incorrect assertions, presents a critical barrier to the adoption of these technologies in high-stakes environments where information integrity is non-negotiable.<n>This empirical evaluation explores the effectiveness of Retrieval-Augmented Generation (RAG) architectures in mitigating these risks by grounding generative outputs in authoritative document context.
  arXiv  Detail & Related papers  (2026-01-21T20:52:48Z)
• Eval Factsheets: A Structured Framework for Documenting AI Evaluations [18.275882528334794]
  We introduce Eval Factsheets, a framework for documenting AI system evaluations.<n>Our framework organizes evaluation characteristics across five fundamental dimensions.<n>We demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms.
  arXiv  Detail & Related papers  (2025-12-03T18:46:50Z)
• ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search [69.60882125603133]
  We present ReliabilityRAG, a framework for adversarial robustness that explicitly leverages reliability information of retrieved documents.<n>Our work is a significant step towards more effective, provably robust defenses against retrieved corpus corruption in RAG.
  arXiv  Detail & Related papers  (2025-09-27T22:36:42Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• AI-Supported Platform for System Monitoring and Decision-Making in Nuclear Waste Management with Large Language Models [1.6795461001108096]
  This paper presents a multi-agent Retrieval-Augmented Generation (RAG) system that integrates large language models (LLMs) with document retrieval mechanisms.<n>The system ensures regulatory decisions remain factually grounded, dynamically adapting to evolving regulatory frameworks.
  arXiv  Detail & Related papers  (2025-05-27T20:29:53Z)
• CReSt: A Comprehensive Benchmark for Retrieval-Augmented Generation with Complex Reasoning over Structured Documents [6.359764486371197]
  Large Language Models (LLMs) have made substantial progress in recent years, yet evaluating their capabilities in practical Retrieval-Augmented Generation (RAG) scenarios remains challenging.<n>We present CReSt (A Comprehensive Benchmark for Retrieval-Augmented Generation with Complex Reasoning over Structured Documents), a benchmark designed to assess these key dimensions holistically.<n>CReSt comprises 2,245 human-annotated examples in English and Korean, designed to capture practical RAG scenarios that require complex reasoning over structured documents.
  arXiv  Detail & Related papers  (2025-05-23T05:56:25Z)
• Revisiting Model Inversion Evaluation: From Misleading Standards to Reliable Privacy Assessment [63.07424521895492]
  Model Inversion (MI) attacks aim to reconstruct information from private training data by exploiting access to machine learning models T.<n>The standard evaluation framework for such attacks relies on an evaluation model E, trained under the same task design as T.<n>This framework has become the de facto standard for assessing progress in MI research, used across nearly all recent MI attacks and defenses without question.
  arXiv  Detail & Related papers  (2025-05-06T13:32:12Z)
• Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation [52.83870601473094]
  Embodied agents exhibit immense potential across a multitude of domains.<n>Existing research predominantly concentrates on the security of general large language models.<n>This paper introduces a novel input moderation framework, meticulously designed to safeguard embodied agents.
  arXiv  Detail & Related papers  (2025-04-22T08:34:35Z)
• Structured Legal Document Generation in India: A Model-Agnostic Wrapper Approach with VidhikDastaavej [5.790242888372048]
  We introduce VidhikDastaavej, a novel, anonymized dataset of private legal documents.<n>We develop NyayaShilp, a fine-tuned legal document generation model specifically adapted to Indian legal texts.
  arXiv  Detail & Related papers  (2025-04-04T14:41:50Z)
• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection.<n>Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains.<n>We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework [61.38174427966444]
  Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios.<n>Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models.<n>We propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses.
  arXiv  Detail & Related papers  (2025-02-26T06:31:45Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain.<n>Our benchmark is characterized by its multi-dimensional evaluation framework.<n>Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validation [6.544757635738911]
  Retrieval-augmented generation (RAG) is an umbrella of different components, design decisions, and domain-specific adaptations. There is currently no generally accepted methodology for RAG evaluation despite a growing interest in this technology. We propose a first blueprint of a methodology for a sound and reliable evaluation of RAG systems.
  arXiv  Detail & Related papers  (2024-10-11T13:36:13Z)
• Improving Legal Entity Recognition Using a Hybrid Transformer Model and Semantic Filtering Approach [0.0]
  This paper proposes a novel hybrid model that enhances the accuracy and precision of Legal-BERT, a transformer model fine-tuned for legal text processing. We evaluate the model on a dataset of 15,000 annotated legal documents, achieving an F1 score of 93.4%, demonstrating significant improvements in precision and recall over previous methods.
  arXiv  Detail & Related papers  (2024-10-11T04:51:28Z)
• VERA: Validation and Evaluation of Retrieval-Augmented Systems [5.709401805125129]
  VERA is a framework designed to enhance the transparency and reliability of outputs from large language models (LLMs) We show how VERA can strengthen decision-making processes and trust in AI applications.
  arXiv  Detail & Related papers  (2024-08-16T21:59:59Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.