WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2505.08643v1
• Date: Tue, 13 May 2025 15:02:54 GMT
• Title: WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation
• Abstract summary: We introduce WixQA, a benchmark suite featuring QA datasets precisely grounded in the released knowledge base (KB) corpus.<n>WixQA includes three distinct QA datasets derived from Wix.com customer support interactions.<n>We release the KB snapshot alongside the datasets under MIT license and provide comprehensive baseline results.
• Score: 0.565395466029518
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-Augmented Generation (RAG) is a cornerstone of modern question answering (QA) systems, enabling grounded answers based on external knowledge. Although recent progress has been driven by open-domain datasets, enterprise QA systems need datasets that mirror the concrete, domain-specific issues users raise in day-to-day support scenarios. Critically, evaluating end-to-end RAG systems requires benchmarks comprising not only question--answer pairs but also the specific knowledge base (KB) snapshot from which answers were derived. To address this need, we introduce WixQA, a benchmark suite featuring QA datasets precisely grounded in the released KB corpus, enabling holistic evaluation of retrieval and generation components. WixQA includes three distinct QA datasets derived from Wix.com customer support interactions and grounded in a snapshot of the public Wix Help Center KB: (i) WixQA-ExpertWritten, 200 real user queries with expert-authored, multi-step answers; (ii) WixQA-Simulated, 200 expert-validated QA pairs distilled from user dialogues; and (iii) WixQA-Synthetic, 6,222 LLM-generated QA pairs, with one pair systematically derived from each article in the knowledge base. We release the KB snapshot alongside the datasets under MIT license and provide comprehensive baseline results, forming a unique benchmark for evaluating enterprise RAG systems in realistic enterprise environments.

Related papers

• Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation [21.656551146954587]
  Existing datasets challenge current RAG systems on Wikipedia-based knowledge.<n>A comprehensive evaluation of RAG system quality requires both multi-hop queries and unanswerable questions.<n>This paper introduces TRIAD, a three-stage automated dataset generation approach.
  arXiv  Detail & Related papers  (2026-08-21T18:46:55Z)
• VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering [53.662676566188175]
  A key bottleneck lies in the lack of public, large-scale, high-quality Scientific Visual Question Answering (SVQA) datasets.<n>We propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context.<n>We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types.
  arXiv  Detail & Related papers  (2025-11-25T04:14:52Z)
• The benefits of query-based KGQA systems for complex and temporal questions in LLM era [55.20230501807337]
  Large language models excel in question-answering (QA) yet still struggle with multi-hop reasoning and temporal questions.<n> Query-based knowledge graph QA (KGQA) offers a modular alternative by generating executable queries instead of direct answers.<n>We explore multi-stage query-based framework for WikiData QA, proposing multi-stage approach that enhances performance on challenging multi-hop and temporal benchmarks.
  arXiv  Detail & Related papers  (2025-07-16T06:41:03Z)
• Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking [56.27361644734853]
  Knowledge Graph Question Answering systems rely on high-quality benchmarks to evaluate complex multi-hop reasoning.<n>Despite their widespread use, popular datasets such as WebQSP and CWQ suffer from critical quality issues.<n>We introduce KGQAGen, an LLM-in-the-loop framework that systematically resolves these pitfalls.<n>Our findings advocate for more rigorous benchmark construction and position KGQAGen as a scalable framework for advancing KGQA evaluation.
  arXiv  Detail & Related papers  (2025-05-29T14:44:52Z)
• AMAQA: A Metadata-based QA Dataset for RAG Systems [7.882922366782987]
  We present AMAQA, a new open-access QA dataset designed to evaluate tasks combining text and metadata.<n>AMAQA includes about 1.1 million English messages collected from 26 public Telegram groups.<n>We show that leveraging metadata boosts accuracy from 0.12 to 0.61, highlighting the value of structured context.
  arXiv  Detail & Related papers  (2025-05-19T08:59:08Z)
• Revolutionizing Database Q&A with Large Language Models: Comprehensive Benchmark and Evaluation [43.217701432032484]
  We introduce DQABench, the first comprehensive database QA benchmark for Large Language Models (LLMs)<n> DQABench features an innovative LLM-based method to automate the generation, cleaning, and rewriting of evaluation dataset, resulting in over 200,000 QA pairs in English and Chinese, separately.<n> Furthermore, we propose a comprehensive database QA testbed DQATestbed. This testbed is highly modular and scalable, with basic and advanced components such as Question Classification Routing (QCR), RAG, TIG, and Prompt template Engineering (PTE)
  arXiv  Detail & Related papers  (2024-09-05T13:45:42Z)
• Is the House Ready For Sleeptime? Generating and Evaluating Situational Queries for Embodied Question Answering [48.43453390717167]
  We present and tackle the problem of Embodied Question Answering with Situational Queries (S-EQA) in a household environment.<n>Unlike prior EQA work, situational queries require the agent to correctly identify multiple object-states and reach a consensus on their states for an answer.<n>We introduce a novel Prompt-Generate-Evaluate scheme that wraps around an LLM's output to generate unique situational queries and corresponding consensus object information.
  arXiv  Detail & Related papers  (2024-05-08T00:45:20Z)
• QASnowball: An Iterative Bootstrapping Framework for High-Quality Question-Answering Data Generation [67.27999343730224]
  We introduce an iterative bootstrapping framework for QA data augmentation (named QASnowball) QASnowball can iteratively generate large-scale high-quality QA data based on a seed set of supervised examples. We conduct experiments in the high-resource English scenario and the medium-resource Chinese scenario, and the experimental results show that the data generated by QASnowball can facilitate QA models.
  arXiv  Detail & Related papers  (2023-09-19T05:20:36Z)
• ADMUS: A Progressive Question Answering Framework Adaptable to Multiple Knowledge Sources [9.484792817869671]
  We present ADMUS, a progressive knowledge base question answering framework designed to accommodate a wide variety of datasets. Our framework supports the seamless integration of new datasets with minimal effort, only requiring creating a dataset-related micro-service at a negligible cost.
  arXiv  Detail & Related papers  (2023-08-09T08:46:39Z)
• A Benchmark for Generalizable and Interpretable Temporal Question Answering over Knowledge Bases [67.33560134350427]
  TempQA-WD is a benchmark dataset for temporal reasoning. It is based on Wikidata, which is the most frequently curated, openly available knowledge base.
  arXiv  Detail & Related papers  (2022-01-15T08:49:09Z)
• OpenQA: Hybrid QA System Relying on Structured Knowledge Base as well as Non-structured Data [15.585969737147892]
  We propose an intelligent question-answering system based on structured KB and unstructured data, called OpenQA. We integrate KBQA structured question answering based on semantic parsing and deep representation learning, and two-stage unstructured question answering based on retrieval and neural machine reading comprehension into OpenQA.
  arXiv  Detail & Related papers  (2021-12-31T09:15:39Z)
• SYGMA: System for Generalizable Modular Question Answering OverKnowledge Bases [57.89642289610301]
  We present SYGMA, a modular approach facilitating general-izability across multiple knowledge bases and multiple rea-soning types. We demonstrate effectiveness of our system by evaluating on datasets belonging to two distinct knowledge bases,DBpedia and Wikidata.
  arXiv  Detail & Related papers  (2021-09-28T01:57:56Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.