HETERQA: Benchmarking Record Retrieval over Multiple Heterogeneous Sources

• URL: http://arxiv.org/abs/2607.03028v1
• Date: Fri, 03 Jul 2026 07:18:33 GMT
• Title: HETERQA: Benchmarking Record Retrieval over Multiple Heterogeneous Sources
• Abstract summary: We introduce HETERQA, a benchmark for record retrieval over five heterogeneous sources.<n>HeterQA instantiates this setting with Yelp business records, each of which is grounded by multiple sources.<n>We validate the benchmark through contradiction detection and human validation, and further evaluate sparse, dense, hybrid, late-interaction, and agentic retrievers.
• Score: 15.327691815902483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In emerging systems (e.g., social media and e-commerce platforms), data records are often drawn from heterogeneous sources, such as relational tables, text documents, image repositories, spatial databases, and knowledge graphs. Accordingly, retrieving target records for question-answering (QA) tasks requires us to jointly exploit these heterogeneous sources. However, most existing benchmarks are constructed from individual sources, and only a very few recent benchmarks have considered two or three sources. To alleviate this issue, we introduce HETERQA, a comprehensive benchmark with 857 QA pairs for record retrieval over five heterogeneous sources. HETERQA instantiates this setting with Yelp business records, each of which is grounded by multiple sources. We build HETERQA in an answer-driven manner: candidate records are first initialized with record-field constraints, then enriched through heterogeneous sources, and finally cross-verified across required sources before the natural-language question is retained. We validate the benchmark through contradiction detection and human validation, and further evaluate sparse, dense, hybrid, late-interaction, and agentic retrievers under the same metrics. The results show that HETERQA is challenging: hybrid retrieval achieves the strongest Recall@10, Self-RAG achieves the best MRR@10, and all evaluated methods remain far from saturating the benchmark. These findings indicate that HETERQA provides an effective testbed for record retrieval over heterogeneous sources and leaves substantial room for future retrieval methods. The benchmark dataset and source code are publicly available at https://huggingface.co/datasets/hanchang02/HeterQA and https://github.com/hanchang02/HeterQA, respectively.

Related papers

• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• Source-Aware Reranking for Retrieval-Augmented Generation: A Reliability Prior Approach [0.0]
  This work evaluates a simple and interpretable modification to RAG retrieval ranking that incorporates provenance-informed source reliability priors.<n>The framework is evaluated against a similarity-only baseline on a 120-document health-domain corpus.<n> source-aware reranking improves Precision@5 from 0.48 to 0.72 and reduces average adversarial document retrieval.
  arXiv  Detail & Related papers  (2026-06-08T14:09:04Z)
• Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
  Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
  arXiv  Detail & Related papers  (2026-03-09T18:20:20Z)
• iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics [9.25340189071758]
  We present iAgentBench, a dynamic ODQA benchmark for cross-source sensemaking.<n>iAgentBench draws seed topics from real-world attention signals and uses common user intent patterns to construct user-like questions.<n>Each instance is released with traceable evidence and auditable intermediate artifacts that support contamination checks.
  arXiv  Detail & Related papers  (2026-03-04T22:40:08Z)
• HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers [9.867003380698238]
  Leveraging labeled and unlabeled data may provide complementary benefits in retrieval augmented generation.<n>We propose a method to combine evidence from these heterogeneous sources.<n>We evaluate our approach on the fact verification task, demonstrating that it consistently improves over the best-performing individual ranker or source.
  arXiv  Detail & Related papers  (2025-09-02T21:03:40Z)
• MSRS: Evaluating Multi-Source Retrieval-Augmented Generation [51.717139132190574]
  Many real-world applications demand the ability to integrate and summarize information scattered across multiple sources.<n>We present a scalable framework for constructing evaluation benchmarks that challenge RAG systems to integrate information across distinct sources.
  arXiv  Detail & Related papers  (2025-08-28T14:59:55Z)
• Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval [22.33550491040999]
  RAG grounds large language models in external evidence, yet it still falters when answers must be pieced together across semantically distant documents.<n>We build two plug-and-play retrievers: StatementGraphRAG and TopicGraphRAG.<n>Our methods outperform naive chunk-based RAG achieving an average relative improvement of 23.1% in retrieval recall and correctness.
  arXiv  Detail & Related papers  (2025-06-09T17:58:35Z)
• Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset [8.867885891794877]
  We introduce Hybrid-SQuAD, a novel large-scale Scholarly Question Answering dataset.<n>The dataset consists of 10.5K question-answer pairs generated by a large language model.<n>We propose a RAG-based baseline hybrid QA model, achieving an exact match score of 69.65 on the Hybrid-SQuAD test set.
  arXiv  Detail & Related papers  (2024-12-03T19:37:00Z)
• Retrieval-Augmented Generation with Estimation of Source Reliability [28.70905685371307]
  Reliability-Aware RAG (RA-RAG) is a new multi-source RAG framework that estimates the reliability of sources.<n> RA-RAG first estimates source reliability by cross-checking information across multiple sources.<n>It then retrieves documents from the top-$kappa$ reliable and relevant sources and aggregates their information using weighted majority voting (WMV)
  arXiv  Detail & Related papers  (2024-10-30T12:09:29Z)
• Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge [82.5582220249183]
  We propose a novel open-domain question answering (ODQA) framework for answering single/multi-hop questions across heterogeneous knowledge sources. Unlike previous methods that solely rely on the retriever for gathering all evidence in isolation, our intermediary performs a chain of reasoning over the retrieved set. Our system achieves competitive performance on two ODQA datasets, OTT-QA and NQ, against tables and passages from Wikipedia.
  arXiv  Detail & Related papers  (2022-10-22T03:21:32Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.