Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking

• URL: http://arxiv.org/abs/2605.12028v1
• Date: Tue, 12 May 2026 12:13:45 GMT
• Title: Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking
• Abstract summary: We describe our system for SemEval-2026 Task 8 (MTRAGEval), participating in Task A (Retrieval) across four English-language domains.<n>Our approach employs a three-stage pipeline: (1) query rewriting via a LoRA-fine-tuned Qwen 2.5 7B model that transforms context-dependent follow-up questions into standalone queries, (2) hybrid BM25 and dense retrieval combined through Reciprocal Rank Fusion, and (3) cross-encoder reranking with BGE-reranker-v2-m3.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We describe our system for SemEval-2026 Task 8 (MTRAGEval), participating in Task A (Retrieval) across four English-language domains. Our approach employs a three-stage pipeline: (1) query rewriting via a LoRA-fine-tuned Qwen 2.5 7B model that transforms context-dependent follow-up questions into standalone queries, (2) hybrid BM25 and dense retrieval combined through Reciprocal Rank Fusion, and (3) cross-encoder reranking with BGE-reranker-v2-m3. On the official test set, the system achieves nDCG@5 of 0.531, ranking 8th out of 38 participating systems and 10.7% above the organizer baseline. Development comparisons reveal that domain-specific temperature tuning for query generation, where technical domains benefit from deterministic decoding and general domains from controlled randomness, provides consistent gains, while more complex strategies such as domain-aware prompting and multi-query expansion degrade performance.

Related papers

• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG [2.9721401411878254]
  Multi-turn retrieval-augmented generation (RAG) is challenging due to evolving user intent, conversational noise, and strict context limits.<n>We propose a training-free hybrid retrieval pipeline for SemEval-2026 Task 8 that combines dense and sparse retrieval with controlled query rewriting and cross-encoder reranking.
  arXiv  Detail & Related papers  (2026-06-05T07:28:01Z)
• EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design [0.4499833362998488]
  We introduce a benchmark suite with three evaluation dimensions.<n>We present EngiAI, a Multi-Agent System (MAS) implementation built on LangGraph.
  arXiv  Detail & Related papers  (2026-05-19T12:12:09Z)
• MARQUIS: A Three-Stage Pipeline for Video Retrieval-Augmented Generation [64.33427079838602]
  Retrieval-augmented generation from videos requires systems to retrieve relevant audiovisual evidence from large corpora and synthesize it into coherent, attributed text.<n>We present MARQUIS: a three-stage pipeline that addresses these limitations through query expansion, fusion, and reranking.<n>For article generation, ITER-QA-BASE improves average human score from 3.09 to 3.83 over the CAG baseline, while MARQUIS-RLM achieves a human score of 3.30 and the strongest citation recall among non-QA systems.
  arXiv  Detail & Related papers  (2026-05-17T20:19:04Z)
• Adaptive ToR: Complexity-Aware Tree-Based Retrieval for Pareto-Optimal Multi-Intent NLU [0.15293427903448018]
  Multi-intent natural language understanding requires retrieval systems that simultaneously achieve high accuracy and computational efficiency.<n>This paper proposes Adaptive Tree-of-Retrieval (Adaptive ToR), a complexity-aware retrieval architecture that dynamically configures retrieval topology based on query characteristics.
  arXiv  Detail & Related papers  (2026-04-27T09:24:10Z)
• From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents [0.0]
  We benchmark ten retrieval strategies spanning sparse, dense, hybrid fusion, cross-encoder reranking, query expansion, index augmentation, and adaptive retrieval.<n>We evaluate retrieval quality via Recall@k, MRR, and nDCG, and end-to-end generation quality via Number Match.
  arXiv  Detail & Related papers  (2026-04-02T07:53:40Z)
• Test-Time Strategies for More Efficient and Accurate Agentic RAG [58.44913384057518]
  Retrieval-Augmented Generation (RAG) systems face challenges with complex, multihop questions.<n>Such approaches can introduce inefficiencies, including repetitive retrieval of previously processed information.<n>We investigate test-time modifications to the Search-R1 pipeline to mitigate these identified shortcomings.
  arXiv  Detail & Related papers  (2026-03-12T19:18:59Z)
• AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations [14.20669481486209]
  We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval)<n>We address all three subtasks of multi-turn retrieval-augmented generation: passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C)<n>Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, and (ii) a multistage generation pipeline.
  arXiv  Detail & Related papers  (2026-03-11T08:28:31Z)
• StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems [0.0]
  StratRAG is an evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems.<n>It comprises 2,200 examples across three question types -- bridge, comparison, and yes-no -- each paired with a pool of 15 candidate documents.<n>We benchmark three retrieval strategies -- BM25, dense retrieval, and hybrid fusion.
  arXiv  Detail & Related papers  (2026-03-06T20:08:06Z)
• FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understanding [19.90863270947963]
  Existing language models struggle with Standard Operating Procedures ( SOPs)<n>We propose FM SO.P, solving these challenges through two novelties.<n>First, we introduce progressive task mixtures that build capabilities by stages across three task types with cumulative data.<n>Second, we propose an automatic multi-agent evaluation system consisting of three agents that adaptively generate rubrics, stratified test sets, and scoring, adapting to domains.
  arXiv  Detail & Related papers  (2026-02-10T02:09:48Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization [86.57743637722596]
  Large Language Model (LLM)-based web agents demonstrate strong performance on knowledge-intensive tasks.<n>Complex queries involving multiple entities, intertwined relationships, and high uncertainty demand extensive search cycles.<n>We introduce ReSum, a novel paradigm that enables indefinite exploration through periodic context summarization.
  arXiv  Detail & Related papers  (2025-09-16T17:57:22Z)
• Dense Passage Retrieval for Open-Domain Question Answering [49.028342823838486]
  We show that retrieval can be practically implemented using dense representations alone. Our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy.
  arXiv  Detail & Related papers  (2020-04-10T04:53:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.