Building Efficient and Effective OpenQA Systems for Low-Resource Languages

• URL: http://arxiv.org/abs/2401.03590v2
• Date: Wed, 5 Jun 2024 03:13:31 GMT
• Title: Building Efficient and Effective OpenQA Systems for Low-Resource Languages
• Authors: Emrah Budur, Rıza Özçelik, Dilara Soylu, Omar Khattab, Tunga Güngör, Christopher Potts,
• Abstract summary: We show that effective, low-cost OpenQA systems can be developed for low-resource contexts. Key ingredients are weak supervision using machine-translated labeled datasets and a relevant unstructured knowledge source. We present SQuAD-TR, a machine translation of SQuAD2.0, and we build our OpenQA system by adapting ColBERT-QA and retraining it over Turkish resources.
• Score: 17.64851283209797
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Question answering (QA) is the task of answering questions posed in natural language with free-form natural language answers extracted from a given passage. In the OpenQA variant, only a question text is given, and the system must retrieve relevant passages from an unstructured knowledge source and use them to provide answers, which is the case in the mainstream QA systems on the Web. QA systems currently are mostly limited to the English language due to the lack of large-scale labeled QA datasets in non-English languages. In this paper, we show that effective, low-cost OpenQA systems can be developed for low-resource contexts. The key ingredients are (1) weak supervision using machine-translated labeled datasets and (2) a relevant unstructured knowledge source in the target language context. Furthermore, we show that only a few hundred gold assessment examples are needed to reliably evaluate these systems. We apply our method to Turkish as a challenging case study, since English and Turkish are typologically very distinct and Turkish has limited resources for QA. We present SQuAD-TR, a machine translation of SQuAD2.0, and we build our OpenQA system by adapting ColBERT-QA and retraining it over Turkish resources and SQuAD-TR using two versions of Wikipedia dumps spanning two years. We obtain a performance improvement of 24-32% in the Exact Match (EM) score and 22-29% in the F1 score compared to the BM25-based and DPR-based baseline QA reader models. Our results show that SQuAD-TR makes OpenQA feasible for Turkish, which we hope encourages researchers to build OpenQA systems in other low-resource languages. We make all the code, models, and the dataset publicly available at https://github.com/boun-tabi/SQuAD-TR.

Related papers

• Grounding-IQA: Multimodal Language Grounding Model for Image Quality Assessment [69.07445098168344]
  We introduce a new image quality assessment (IQA) task paradigm, grounding-IQA. Grounding-IQA comprises two subtasks: grounding-IQA-description (GIQA-DES) and visual question answering (GIQA-VQA) To realize grounding-IQA, we construct a corresponding dataset, GIQA-160K, through our proposed automated annotation pipeline. Experiments demonstrate that our proposed task paradigm, dataset, and benchmark facilitate the more fine-grained IQA application.
  arXiv  Detail & Related papers  (2024-11-26T09:03:16Z)
• UQA: Corpus for Urdu Question Answering [3.979019316355144]
  This paper introduces UQA, a novel dataset for question answering and text comprehension in Urdu. UQA is generated by translating the Stanford Question Answering dataset (SQuAD2.0), a large-scale English QA dataset. The paper describes the process of selecting and evaluating the best translation model among two candidates: Google Translator and Seamless M4T.
  arXiv  Detail & Related papers  (2024-05-02T16:44:31Z)
• Can a Multichoice Dataset be Repurposed for Extractive Question Answering? [52.28197971066953]
  We repurposed the Belebele dataset (Bandarkar et al., 2023), which was designed for multiple-choice question answering (MCQA) We present annotation guidelines and a parallel EQA dataset for English and Modern Standard Arabic (MSA). Our aim is to enable others to adapt our approach for the 120+ other language variants in Belebele, many of which are deemed under-resourced.
  arXiv  Detail & Related papers  (2024-04-26T11:46:05Z)
• EuSQuAD: Automatically Translated and Aligned SQuAD2.0 for Basque [0.4499833362998487]
  This work presents EuSQuAD, the first initiative dedicated to automatically translating and aligning SQuAD2.0 into Basque. We demonstrate EuSQuAD's value through extensive qualitative analysis and QA experiments supported with EuSQuAD as training data.
  arXiv  Detail & Related papers  (2024-04-18T13:31:57Z)
• Evaluating and Modeling Attribution for Cross-Lingual Question Answering [80.4807682093432]
  This work is the first to study attribution for cross-lingual question answering. We collect data in 5 languages to assess the attribution level of a state-of-the-art cross-lingual QA system. We find that a substantial portion of the answers is not attributable to any retrieved passages.
  arXiv  Detail & Related papers  (2023-05-23T17:57:46Z)
• Generative Language Models for Paragraph-Level Question Generation [79.31199020420827]
  Powerful generative models have led to recent progress in question generation (QG) It is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. We introduce QG-Bench, a benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting.
  arXiv  Detail & Related papers  (2022-10-08T10:24:39Z)
• XLMRQA: Open-Domain Question Answering on Vietnamese Wikipedia-based Textual Knowledge Source [2.348805691644086]
  This paper presents XLMRQA, the first Vietnamese QA system using a supervised transformer-based reader on the Wikipedia-based textual knowledge source. From the results obtained on the three systems, we analyze the influence of question types on the performance of the QA systems.
  arXiv  Detail & Related papers  (2022-04-14T14:54:33Z)
• DUAL: Textless Spoken Question Answering with Speech Discrete Unit Adaptive Learning [66.71308154398176]
  Spoken Question Answering (SQA) has gained research attention and made remarkable progress in recent years. Existing SQA methods rely on Automatic Speech Recognition (ASR) transcripts, which are time and cost-prohibitive to collect. This work proposes an ASR transcript-free SQA framework named Discrete Unit Adaptive Learning (DUAL), which leverages unlabeled data for pre-training and is fine-tuned by the SQA downstream task.
  arXiv  Detail & Related papers  (2022-03-09T17:46:22Z)
• Pre-trained Transformer-Based Approach for Arabic Question Answering : A Comparative Study [0.5801044612920815]
  We evaluate the state-of-the-art pre-trained transformers models for Arabic QA using four reading comprehension datasets. We fine-tuned and compared the performance of the AraBERTv2-base model, AraBERTv0.2-large model, and AraELECTRA model.
  arXiv  Detail & Related papers  (2021-11-10T12:33:18Z)
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering [62.88322725956294]
  We review the latest research trends in OpenQA, with particular attention to systems that incorporate neural MRC techniques. We introduce modern OpenQA architecture named Retriever-Reader'' and analyze the various systems that follow this architecture. We then discuss key challenges to developing OpenQA systems and offer an analysis of benchmarks that are commonly used.
  arXiv  Detail & Related papers  (2021-01-04T04:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.