Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervision

• URL: http://arxiv.org/abs/2402.16508v3
• Date: Wed, 02 Oct 2024 07:51:47 GMT
• Title: Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervision
• Authors: Fan Jiang, Tom Drummond, Trevor Cohn,
• Abstract summary: Cross-lingual open domain question answering is a complex problem. We show that CLQA can be addressed using a single encoder-decoder model. We propose a self-supervised method based on exploiting the cross-lingual link structure within Wikipedia.
• Score: 44.04243892727856
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-lingual open domain question answering (CLQA) is a complex problem, comprising cross-lingual retrieval from a multilingual knowledge base, followed by answer generation in the query language. Both steps are usually tackled by separate models, requiring substantial annotated datasets, and typically auxiliary resources, like machine translation systems to bridge between languages. In this paper, we show that CLQA can be addressed using a single encoder-decoder model. To effectively train this model, we propose a self-supervised method based on exploiting the cross-lingual link structure within Wikipedia. We demonstrate how linked Wikipedia pages can be used to synthesise supervisory signals for cross-lingual retrieval, through a form of cloze query, and generate more natural questions to supervise answer generation. Together, we show our approach, \texttt{CLASS}, outperforms comparable methods on both supervised and zero-shot language adaptation settings, including those using machine translation.

Related papers

• Few-Shot Multilingual Open-Domain QA from 5 Examples [44.04243892727856]
  We introduce a emphfew-shot learning approach to synthesise large-scale multilingual data from large language models (LLMs) Our method begins with large-scale self-supervised pre-training using WikiData, followed by training on high-quality synthetic multilingual data generated by prompting LLMs with few-shot supervision. The final model, textscFsModQA, significantly outperforms existing few-shot and supervised baselines in MLODQA and cross-lingual and monolingual retrieval.
  arXiv  Detail & Related papers  (2025-02-27T03:24:57Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, effectively being crosslingual? This study evaluates six state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale [53.92008514395125]
  PAXQA (Projecting annotations for cross-lingual (x) QA) decomposes cross-lingual QA into two stages. We propose a novel use of lexically-constrained machine translation, in which constrained entities are extracted from the parallel bitexts. We show that models fine-tuned on these datasets outperform prior synthetic data generation models over several extractive QA datasets.
  arXiv  Detail & Related papers  (2023-04-24T15:46:26Z)
• UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph [89.98762327725112]
  Multi-hop Question Answering over Knowledge Graph(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question. We propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning.
  arXiv  Detail & Related papers  (2022-12-02T04:08:09Z)
• XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing [70.40401197026925]
  In-context learning using large language models has recently shown surprising results for semantic parsing tasks. This work introduces the XRICL framework, which learns to retrieve relevant English exemplars for a given query. We also include global translation exemplars for a target language to facilitate the translation process for large language models.
  arXiv  Detail & Related papers  (2022-10-25T01:33:49Z)
• ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System [16.89747171947662]
  This paper introduces our proposed system for the MIA Shared Task on Cross-lingual Open-retrieval Question Answering (COQA) In this challenging scenario, given an input question the system has to gather evidence documents from a multilingual pool and generate an answer in the language of the question. We devised several approaches combining different model variants for three main components: Data Augmentation, Passage Retrieval, and Answer Generation.
  arXiv  Detail & Related papers  (2022-05-30T10:31:08Z)
• Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering [20.4489424966613]
  We investigate the capabilities of multilingually pre-trained language models on cross-lingual question answering systems. We find that explicitly aligning the representations across languages with a post-hoc fine-tuning step generally leads to improved performance.
  arXiv  Detail & Related papers  (2021-09-24T15:32:45Z)
• One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval [39.061900747689094]
  CORA is a Cross-lingual Open-Retrieval Answer Generation model. It can answer questions across many languages even when language-specific annotated data or knowledge sources are unavailable.
  arXiv  Detail & Related papers  (2021-07-26T06:02:54Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.