AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

• URL: http://arxiv.org/abs/2603.10524v1
• Date: Wed, 11 Mar 2026 08:28:31 GMT
• Title: AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations
• Abstract summary: We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval)<n>We address all three subtasks of multi-turn retrieval-augmented generation: passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C)<n>Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, and (ii) a multistage generation pipeline.
• Score: 14.20669481486209
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval), addressing all three subtasks of multi-turn retrieval-augmented generation: passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C). Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, where five complementary LLM-based query reformulations are issued to a single corpus-aligned sparse retriever and fused via variance-aware nested Reciprocal Rank Fusion; and (ii) a multistage generation pipeline that decomposes grounded generation into evidence span extraction, dual-candidate drafting, and calibrated multi-judge selection. Our system ranks 1st in Task A (nDCG@5: 0.5776, +20.5% over the strongest baseline) and 2nd in Task B (HM: 0.7698). Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than retrieval coverage-is the primary bottleneck in end-to-end performance.

Related papers

• Generative Universal Multimodal Retrieval with Dual-role Identifiers [1.0981910170846492]
  We propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers.<n>DrIG consistently outperforms state-of-the-art generative multimodal baselines across diverse tasks, while hybrid reranking achieves a favorable efficiency-effectiveness trade-off against strong dense retrievers.
  arXiv  Detail & Related papers  (2026-08-13T09:10:09Z)
• Hierarchical Evidence-Driven Reasoning for Long Document Understanding [95.51688464037096]
  We introduce HIEVI-RAG, a hierarchical, evidence-driven multimodal RAG framework for closed-domain document understanding.<n>Our framework significantly outperforms existing open-source baselines and exceeds the strongest reported baseline by an average of 8.05% in accuracy.
  arXiv  Detail & Related papers  (2026-07-06T03:08:28Z)
• 5ting at SemEval-2026 Task 8: Strong End-to-End Multi-Turn RAG via LLM-Based Reranking and Faithfulness Control [0.0]
  We introduce 5ting, our system for the SemEval2026 Task 8 (MTRAGEval)<n>We evaluate multi-turn Retrieval Augmented Generation (RAG) systems.
  arXiv  Detail & Related papers  (2026-06-27T05:13:49Z)
• Retriever Portfolios: A Principled Approach to Adaptive RAG [30.668406304237404]
  We propose a method that automatically selects a small, diverse subset of retrievers from a large pool of candidates, to cover different regions of the target query distribution.<n>Across multiple QA benchmarks, our learned portfolios and router pipeline consistently outperform single-retriever and naive multi-retriever baselines.
  arXiv  Detail & Related papers  (2026-05-29T11:43:05Z)
• R$^3$AG: Retriever Routing for Retrieval-Augmented Generation [59.47703698994575]
  R$3$AG decomposes retriever capability into two learnable dimensions: retrieval quality and generation utility.<n>R$3$AG consistently outperforms both the best individual retrievers and state-of-the-art static routing methods.
  arXiv  Detail & Related papers  (2026-04-22T06:51:20Z)
• Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
  Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
  arXiv  Detail & Related papers  (2026-03-09T18:20:20Z)
• Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG [29.46121429194507]
  Retrieval-augmented generation (RAG) enables large language models to produce evidence-based responses.<n>Existing solutions suffer from objective mismatch between retriever optimization and the goal of RAG pipeline.
  arXiv  Detail & Related papers  (2026-02-03T15:30:14Z)
• MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval [50.30107119622642]
  Large Language Models (LLMs) excel at reasoning and generation but are inherently limited by static pretraining data.<n>Retrieval-Augmented Generation (RAG) addresses this issue by grounding LLMs in external knowledge.<n>MarAG-R1 is a reinforcement-learned multi-tool RAG framework that enables LLMs to dynamically coordinate multiple retrieval mechanisms.
  arXiv  Detail & Related papers  (2025-10-31T15:51:39Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• SIRAG: Towards Stable and Interpretable RAG with A Process-Supervised Multi-Agent Framework [7.37561751991963]
  We propose a process-supervised multi-agent framework to bridge the gap between retriever and generator.<n>The proposed framework is modular and plug-and-play, requiring no modification to the retriever or generator.
  arXiv  Detail & Related papers  (2025-09-17T09:09:28Z)
• MSRS: Evaluating Multi-Source Retrieval-Augmented Generation [51.717139132190574]
  Many real-world applications demand the ability to integrate and summarize information scattered across multiple sources.<n>We present a scalable framework for constructing evaluation benchmarks that challenge RAG systems to integrate information across distinct sources.
  arXiv  Detail & Related papers  (2025-08-28T14:59:55Z)
• RAGentA: Multi-Agent Retrieval-Augmented Generation for Attributed Question Answering [4.224843546370802]
  We present RAGentA, a framework for attributed question answering with large language models (LLMs)<n>With the goal of trustworthy answer generation, RAGentA focuses on optimizing answer correctness, defined by coverage and relevance to the question and faithfulness.<n>Central to the framework is a hybrid retrieval strategy that combines sparse and dense methods, improving Recall@20 by 12.5% compared to the best single retrieval model.
  arXiv  Detail & Related papers  (2025-06-20T13:37:03Z)
• LTRR: Learning To Rank Retrievers for LLMs [53.285436927963865]
  We show that routing-based RAG systems can outperform the best single-retriever-based systems.<n>Performance gains are especially pronounced in models trained with the Answer Correctness (AC) metric.<n>As part of the SIGIR 2025 LiveRAG challenge, our submitted system demonstrated the practical viability of our approach.
  arXiv  Detail & Related papers  (2025-06-16T17:53:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.