FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturing

• URL: http://arxiv.org/abs/2605.26476v1
• Date: Tue, 26 May 2026 02:32:51 GMT
• Title: FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturing
• Abstract summary: We introduce FAB-Bench, an end-to-end framework for adaptive benchmarking of RAG systems in semiconductor manufacturing.<n> FAB-Bench defines six diagnostic metrics measuring factual accuracy, contextual utilization, completeness, retrieval relevance, technical depth, and reasoning consistency.<n>From over 1,300 generated candidates, we curated a high-quality benchmark of 200 query-answer pairs spanning three synthesis strategies.
• Score: 21.340021473068663
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-Augmented Generation (RAG) has become critical for knowledge-intensive applications, yet evaluating its performance in vertical domains remains difficult due to domain complexity, diverse context scales, and heavy reliance on expert assessments that are costly, inconsistent, and non-scalable. We introduce FAB-Bench, an end-to-end framework for adaptive benchmarking of RAG systems in semiconductor manufacturing. FAB-Bench defines six diagnostic metrics measuring factual accuracy, contextual utilization, completeness, retrieval relevance, technical depth, and reasoning consistency. The framework couples retriever diagnostics with generator-level reasoning analysis across context windows of 4K-32K tokens, quantifying how retrieval precision and generative fidelity co-evolve as contextual scope expands. From over 1,300 generated candidates, we curated a high-quality benchmark of 200 query-answer pairs spanning three synthesis strategies: needle-in-haystack, intra-document multi-topic, and cross-document multi-hop. Systematic evaluation across four LLMs and four RAG frameworks reveals three distinct context-scaling behaviors: logarithmic growth, early saturation, and cold-start dynamics, and identifies attention dilution as the primary mechanism behind performance degradation at extreme context lengths. Cross-framework validation on three additional production RAG systems confirms evaluation portability.

Related papers

• Unlocking Biological Workflows for Robust Protein-Text Question Answering: A Dual-Dimensional RAG Framework [60.82334952881798]
  Protein-Text Question Answering (QA) is crucial for interpreting biological sequences through natural language.<n>We propose 2D-ProteinRAG, a novel framework that empowers Large Language Models (LLMs) to operate within the gold-standard biological research workflow.<n>We show that 2D-Protein consistently achieves state-of-the-art performance, outperforming fine-tuned baselines and other RAG methods.
  arXiv  Detail & Related papers  (2026-05-17T05:03:24Z)
• Deepchecks: Evaluating Retrieval-Augmented Generation (RAG) [4.4767889830241705]
  This paper introduces Deepchecks, a comprehensive framework tailored for evaluating RAG applications.<n>By ensuring alignment with application-specific requirements, Deepchecks framework provides a robust foundation for assessing reliability, relevance, and user satisfaction in RAG systems.
  arXiv  Detail & Related papers  (2026-05-14T07:27:50Z)
• Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems [55.04168927059962]
  Reasoning-intensive retrieval aims to surface evidence that supports downstream reasoning rather than merely matching topical similarity.<n>We introduce BRIGHT-Pro, an expert-annotated benchmark that expands each query with multi-aspect gold evidence.<n>We also construct RTriever- Synth, an aspect-decomposed synthetic corpus that generates complementary positives and positive-conditioned hard negatives.
  arXiv  Detail & Related papers  (2026-05-05T17:42:50Z)
• Transforming External Knowledge into Triplets for Enhanced Retrieval in RAG of LLMs [55.78708003681562]
  Retrieval-Augmented Generation (RAG) mitigates hallucination in large language models (LLMs) by incorporating external knowledge during generation.<n>Existing RAG approaches typically retrieve and unstructured text fragments as context.<n>We propose Tri-RAG, a structured triplet-based retrieval framework that improves retrieval efficiency through reasoning-aligned context construction.
  arXiv  Detail & Related papers  (2026-04-14T11:36:29Z)
• Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Framework [2.2782316435989562]
  Performance evaluation of Retrieval-Augmented Generation systems within enterprise environments is governed by multi-dimensional and composite factors.<n>Existing academic benchmarks fail to systematically diagnose these interlocking challenges.<n>This research proposes a multi-dimensional diagnostic framework by defining a four-axis difficulty taxonomy and integrating it into an enterprise RAG benchmark to diagnose potential system weaknesses.
  arXiv  Detail & Related papers  (2026-04-03T02:10:37Z)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome [109.15093810810214]
  MiroEval is a benchmark and evaluation framework for deep research systems.<n>The benchmark comprises 100 tasks grounded in real user needs.<n>The proposed evaluation suite assesses deep research systems along three complementary dimensions.
  arXiv  Detail & Related papers  (2026-03-30T13:16:03Z)
• RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routing [37.7721677767453]
  We introduce RAG-Bench, the first dataset and benchmark designed for adaptive RAG routing.<n>RAG-Bench revisits retrieval from a query-corpus compatibility perspective and standardizes five representative RAG paradigms for systematic evaluation.<n> Experiments with DeepSeek-V3 and LLaMA-3.1-8B demonstrate that no single RAG paradigm is universally optimal.
  arXiv  Detail & Related papers  (2026-01-30T20:38:11Z)
• PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation [15.230902967865925]
  Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge.<n>Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization.<n>Our benchmark provides a reproducible framework for developing more reliable and efficient RAG systems.
  arXiv  Detail & Related papers  (2025-07-23T16:14:08Z)
• MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluation [5.525151548786079]
  Existing RAG benchmarks often overlook query difficulty, leading to inflated performance on simpler questions and unreliable evaluations.<n>We propose MHTS (Multi-Hop Tree Structure), a novel dataset synthesis framework that controls multi-hop reasoning complexity by leveraging a multi-hop tree structure to generate logically connected, multi-chunk queries.
  arXiv  Detail & Related papers  (2025-03-29T06:26:01Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain.<n>Our benchmark is characterized by its multi-dimensional evaluation framework.<n>Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.