FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation

• URL: http://arxiv.org/abs/2510.08945v1
• Date: Fri, 10 Oct 2025 02:51:47 GMT
• Title: FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
• Abstract summary: We introduce a benchmark designed to evaluate RAG pipelines as a whole.<n>We present a dataset of 93 questions designed to evaluate a pipeline's ability to ingest.<n>We find that closed-source pipelines significantly outperform open-source pipelines in both correctness and hallucination metrics.
• Score: 1.6174863821322776
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduce a benchmark designed to evaluate RAG pipelines as a whole, evaluating a pipeline's ability to ingest, retrieve, and reason about several modalities of information, differentiating it from existing benchmarks that focus on particular aspects such as retrieval. We present (1) a small, human-created dataset of 93 questions designed to evaluate a pipeline's ability to ingest textual data, tables, images, and data spread across these modalities in one or more documents; (2) a phrase-level recall metric for correctness; (3) a nearest-neighbor embedding classifier to identify potential pipeline hallucinations; (4) a comparative evaluation of 2 pipelines built with open-source retrieval mechanisms and 4 closed-source foundation models; and (5) a third-party human evaluation of the alignment of our correctness and hallucination metrics. We find that closed-source pipelines significantly outperform open-source pipelines in both correctness and hallucination metrics, with wider performance gaps in questions relying on multimodal and cross-document information. Human evaluation of our metrics showed average agreement of 4.62 for correctness and 4.53 for hallucination detection on a 1-5 Likert scale (5 indicating "strongly agree").

Related papers

• Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation [0.42970700836450487]
  We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions.<n>We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets.
  arXiv  Detail & Related papers  (2026-07-30T19:51:55Z)
• Quantifying the Sources of Instability in LLM-Based Stance Analysis of Public Discourse [4.093009863847175]
  Two distinct sources of instability can arise when preprocessing pipelines produce different outputs from the same input.<n>Using 256 YouTube interviews across 41 public figures from five domains, we compare two speaker-diarization pipelines and two measurement methods.<n>We find that preprocessing pipeline sensitivity is concentrated in speakers with limited video samples.
  arXiv  Detail & Related papers  (2026-07-12T17:15:36Z)
• Hierarchical Evidence-Driven Reasoning for Long Document Understanding [95.51688464037096]
  We introduce HIEVI-RAG, a hierarchical, evidence-driven multimodal RAG framework for closed-domain document understanding.<n>Our framework significantly outperforms existing open-source baselines and exceeds the strongest reported baseline by an average of 8.05% in accuracy.
  arXiv  Detail & Related papers  (2026-07-06T03:08:28Z)
• DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning [64.33887406863899]
  Multimodal deception detection aims to identify deceptive behavior by analyzing audiovisual cues for forensics and security.<n>Existing benchmarks provide only binary labels without intermediate reasoning cues.<n>We construct reasoning datasets with structured cue-level descriptions and reasoning chains.<n>With 1695 samples, it is the largest non-laboratory deception detection dataset.
  arXiv  Detail & Related papers  (2026-03-25T04:06:36Z)
• A Dialectic Pipeline for Improving LLM Robustness [0.0]
  Methods such as fine-tuning on domain-specific data or the training of a separate textitad hoc verifier require demanding computational resources.<n>We propose a dialectic pipeline that preserves LLMs' generalization abilities while improving the quality of its answer via self-dialogue.<n>We find that our proposed dialectic pipeline is able to outperform by significative margins the standard model answers.
  arXiv  Detail & Related papers  (2026-01-28T14:42:49Z)
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• Evaluating Embedding Models and Pipeline Optimization for AI Search Quality [0.0]
  We evaluate the performance of various text embedding models and pipeline configurations for AI-driven search systems.<n>A custom evaluation dataset of 11,975 query-chunk pairs was synthesized from US City Council meeting transcripts.
  arXiv  Detail & Related papers  (2025-11-27T09:09:39Z)
• Retrieval-Augmented Guardrails for AI-Drafted Patient-Portal Messages: Error Taxonomy Construction and Large-Scale Evaluation [5.555479009357263]
  Asynchronous patient-clinician messaging via EHR portals is a growing source of clinician workload.<n>Our contributions are threefold: (1) we introduce a clinically grounded error ontology comprising 5 domains and 59 granular error codes; (2) we develop a retrieval-augmented evaluation pipeline; and (3) we provide a two-stage prompting architecture using DSPy to enable scalable, interpretable, and hierarchical error detection.
  arXiv  Detail & Related papers  (2025-09-26T16:42:43Z)
• DiFaR: Enhancing Multimodal Misinformation Detection with Diverse, Factual, and Relevant Rationales [17.889203383750864]
  DiFaR produces diverse, factual, and relevant rationales to enhance misinformation detection.<n>Experiments on four popular benchmarks demonstrate that DiFaR outperforms four baseline categories by up to 5.9%.<n>Both automatic metrics and human evaluations confirm that DiFaR significantly improves rationale quality across all three dimensions.
  arXiv  Detail & Related papers  (2025-08-14T08:32:31Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain.<n>Our benchmark is characterized by its multi-dimensional evaluation framework.<n>Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification [41.330719056639616]
  We study the entailment verification problem of multi-sentence premises. Modern NLP problems, such as detecting inconsistent model-generated rationales, require complex multi-hop reasoning.
  arXiv  Detail & Related papers  (2024-02-06T04:14:09Z)
• Rethinking Evaluation Metrics of Open-Vocabulary Segmentaion [78.76867266561537]
  The evaluation process still heavily relies on closed-set metrics without considering the similarity between predicted and ground truth categories. To tackle this issue, we first survey eleven similarity measurements between two categorical words. We designed novel evaluation metrics, namely Open mIoU, Open AP, and Open PQ, tailored for three open-vocabulary segmentation tasks.
  arXiv  Detail & Related papers  (2023-11-06T18:59:01Z)
• Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking [56.80065604034095]
  We introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant. To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.
  arXiv  Detail & Related papers  (2022-10-19T16:19:37Z)
• SummEval: Re-evaluating Summarization Evaluation [169.622515287256]
  We re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion. We benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics. We assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset.
  arXiv  Detail & Related papers  (2020-07-24T16:25:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.