Q-NL Verifier: Leveraging Synthetic Data for Robust Knowledge Graph Question Answering

• URL: http://arxiv.org/abs/2503.01385v1
• Date: Mon, 03 Mar 2025 10:28:24 GMT
• Title: Q-NL Verifier: Leveraging Synthetic Data for Robust Knowledge Graph Question Answering
• Authors: Tim Schwabe, Louisa Siebel, Patrik Valach, Maribel Acosta,
• Abstract summary: We present Q-NL Verifier, an approach to generating high-quality synthetic pairs of queries and NL translations.<n>Our approach relies on large language models to generate semantically precise natural language paraphrases of structured queries.<n>Our experiments with the well-known LC-QuAD 2.0 benchmark show that Q-NL Verifier generalizes well to paraphrases from other models and even human-authored translations.
• Score: 0.4499833362998489
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Question answering (QA) requires accurately aligning user questions with structured queries, a process often limited by the scarcity of high-quality query-natural language (Q-NL) pairs. To overcome this, we present Q-NL Verifier, an approach to generating high-quality synthetic pairs of queries and NL translations. Our approach relies on large language models (LLMs) to generate semantically precise natural language paraphrases of structured queries. Building on these synthetic Q-NL pairs, we introduce a learned verifier component that automatically determines whether a generated paraphrase is semantically equivalent to the original query. Our experiments with the well-known LC-QuAD 2.0 benchmark show that Q-NL Verifier generalizes well to paraphrases from other models and even human-authored translations. Our approach strongly aligns with human judgments across varying query complexities and outperforms existing NLP metrics in assessing semantic correctness. We also integrate the verifier into QA pipelines, showing that verifier-filtered synthetic data has significantly higher quality in terms of translation correctness and enhances NL to Q translation accuracy. Lastly, we release an updated version of the LC-QuAD 2.0 benchmark containing our synthetic Q-NL pairs and verifier scores, offering a new resource for robust and scalable QA.

Related papers

• Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation [55.73341401764367]
  We introduce ADSQE, a novel framework for alleviating distribution shift in synthetic QE data.<n>ADSQE uses references, i.e., translation supervision signals, to guide both the generation and annotation processes.<n>Experiments demonstrate that ADSQE outperforms SOTA baselines like COMET in both supervised and unsupervised settings.
  arXiv  Detail & Related papers  (2025-02-27T10:11:53Z)
• Quality-Aware Decoding: Unifying Quality Estimation and Decoding [12.843274390224853]
  We present a novel token-level QE model capable of reliably scoring partial translations.<n>We then present a decoding strategy that integrates the QE model for Quality-Aware decoding.<n>Our approach provides significant benefits in document translation tasks.
  arXiv  Detail & Related papers  (2025-02-12T16:49:52Z)
• When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages [9.138590152838754]
  Segment-level quality estimation (QE) is a challenging cross-lingual language understanding task.<n>We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios.<n>Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models.
  arXiv  Detail & Related papers  (2025-01-08T12:54:05Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [91.981439746404]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation. We first demonstrate the effectiveness of the QASemConsistency methodology for human annotation. We then implement several methods for automatically detecting localized factual inconsistencies.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• QADYNAMICS: Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering [48.25449258017601]
  State-of-the-art approaches fine-tune language models on QA pairs constructed from CommonSense Knowledge Bases. We propose QADYNAMICS, a training dynamics-driven framework for QA diagnostics and refinement.
  arXiv  Detail & Related papers  (2023-10-17T14:27:34Z)
• SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References [73.67707138779245]
  We propose a new evaluation metric: SQuArE (Sentence-level QUestion AnsweRing Evaluation) We evaluate SQuArE on both sentence-level extractive (Answer Selection) and generative (GenQA) QA systems.
  arXiv  Detail & Related papers  (2023-09-21T16:51:30Z)
• PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale [53.92008514395125]
  PAXQA (Projecting annotations for cross-lingual (x) QA) decomposes cross-lingual QA into two stages. We propose a novel use of lexically-constrained machine translation, in which constrained entities are extracted from the parallel bitexts. We show that models fine-tuned on these datasets outperform prior synthetic data generation models over several extractive QA datasets.
  arXiv  Detail & Related papers  (2023-04-24T15:46:26Z)
• Would You Ask it that Way? Measuring and Improving Question Naturalness for Knowledge Graph Question Answering [20.779777536841493]
  Knowledge graph question answering (KGQA) facilitates information access by leveraging structured data without requiring formal query language expertise from the user. We create the IQN-KGQA test collection by sampling questions from existing KGQA datasets and evaluating them with regards to five different aspects of naturalness. We find that some KGQA systems fare worse when presented with more realistic formulations of NL questions.
  arXiv  Detail & Related papers  (2022-05-25T13:32:27Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.