RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

• URL: http://arxiv.org/abs/2506.14412v1
• Date: Tue, 17 Jun 2025 11:14:22 GMT
• Title: RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition
• Authors: Tim Cofala, Oleh Astappiev, William Xion, Hailay Teklehaymanot,
• Abstract summary: The LiveRAG 2025 challenge explores RAG solutions to maximize accuracy on DataMorgana's QA pairs.<n>The challenge provides access to sparse OpenSearch and dense Pinecone indices of the Fineweb 10BT dataset.<n>Our solution achieved a correctness score of 1.13 and a faithfulness score of 0.55, placing fourth in the SIGIR 2025 LiveRAG Challenge.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Retrieval-Augmented Generation (RAG) enriches Large Language Models (LLMs) by combining their internal, parametric knowledge with external, non-parametric sources, with the goal of improving factual correctness and minimizing hallucinations. The LiveRAG 2025 challenge explores RAG solutions to maximize accuracy on DataMorgana's QA pairs, which are composed of single-hop and multi-hop questions. The challenge provides access to sparse OpenSearch and dense Pinecone indices of the Fineweb 10BT dataset. It restricts model use to LLMs with up to 10B parameters and final answer generation with Falcon-3-10B. A judge-LLM assesses the submitted answers along with human evaluators. By exploring distinct retriever combinations and RAG solutions under the challenge conditions, our final solution emerged using InstructRAG in combination with a Pinecone retriever and a BGE reranker. Our solution achieved a correctness score of 1.13 and a faithfulness score of 0.55, placing fourth in the SIGIR 2025 LiveRAG Challenge.

Related papers

• PrismRAG: Boosting RAG Factuality with Distractor Resilience and Strategized Reasoning [57.89188317734747]
  PrismRAG trains the model with distractor-aware QA pairs mixing gold evidence with subtle distractor passages.<n>It instills reasoning-centric habits that make the LLM plan, rationalize, and synthesize without relying on extensive human engineered instructions.
  arXiv  Detail & Related papers  (2025-07-25T00:15:31Z)
• Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge [8.680958290253914]
  We present our submission to the LiveRAG Challenge 2025, which evaluates retrieval-augmented generation (RAG) systems on dynamic test sets.<n>Our final hybrid approach combines sparse (BM25) and dense (E5) retrieval methods.<n>We demonstrate that neural re-ranking with RankLLaMA improves MAP from 0.523 to 0.797 but introduces prohibitive computational costs.
  arXiv  Detail & Related papers  (2025-06-27T21:20:43Z)
• TopClustRAG at SIGIR 2025 LiveRAG Challenge [2.56711111236449]
  TopClustRAG is a retrieval-augmented generation (RAG) system developed for the LiveRAG Challenge.<n>Our system employs a hybrid retrieval strategy combining sparse and dense indices, followed by K-Means clustering to group semantically similar passages.
  arXiv  Detail & Related papers  (2025-06-18T08:24:27Z)
• RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge [4.364909807482374]
  This paper presents the RMIT--ADM+S participation in the SIGIR 2025 LiveRAG Challenge.<n>Our Generation-Retrieval-Augmented Generation (GRAG) approach relies on generating a hypothetical answer that is used in the retrieval phase, alongside the original question.<n>We describe the system architecture and the rationale behind our design choices.
  arXiv  Detail & Related papers  (2025-06-17T13:41:12Z)
• LTRR: Learning To Rank Retrievers for LLMs [53.285436927963865]
  We show that routing-based RAG systems can outperform the best single-retriever-based systems.<n>Performance gains are especially pronounced in models trained with the Answer Correctness (AC) metric.<n>As part of the SIGIR 2025 LiveRAG challenge, our submitted system demonstrated the practical viability of our approach.
  arXiv  Detail & Related papers  (2025-06-16T17:53:18Z)
• Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation [77.10390725623125]
  Long-form question answering (LFQA) presents unique challenges for large language models.<n>RioRAG is a novel reinforcement learning framework that advances long-form RAG through reinforced informativeness optimization.
  arXiv  Detail & Related papers  (2025-05-27T07:34:41Z)
• Chain-of-Retrieval Augmented Generation [72.06205327186069]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer.<n>Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU [3.1787418271023404]
  We designed a Retrieval-Augmented Generation (RAG) system to provide large language models with relevant documents for answering domain-specific questions. We extracted over 1,800 subpages using a greedy scraping strategy and employed a hybrid annotation process, combining manual and Mistral-generated question-answer pairs. Our RAG framework integrates BM25 and FAISS retrievers, enhanced with a reranker for improved document retrieval accuracy.
  arXiv  Detail & Related papers  (2024-11-20T20:10:43Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)
• CRAG -- Comprehensive RAG Benchmark [58.15980697921195]
  Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG) CRAG is a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search.
  arXiv  Detail & Related papers  (2024-06-07T08:43:07Z)
• The Chronicles of RAG: The Retriever, the Chunk and the Generator [0.0]
  This paper presents good practices to implement, optimize, and evaluate RAG for the Brazilian Portuguese language. We explore a diverse set of methods to answer questions about the first Harry Potter book.
  arXiv  Detail & Related papers  (2024-01-15T18:25:18Z)
• Rotation Invariance and Extensive Data Augmentation: a strategy for the Mitosis Domain Generalization (MIDOG) Challenge [1.52292571922932]
  We present the strategy we applied to participate in the MIDOG 2021 competition. The purpose of the competition was to evaluate the generalization of solutions to images acquired with unseen target scanners. We propose a solution based on a combination of state-of-the-art deep learning methods.
  arXiv  Detail & Related papers  (2021-09-02T10:09:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.