Context-Aware Transformer Pre-Training for Answer Sentence Selection

• URL: http://arxiv.org/abs/2305.15358v1
• Date: Wed, 24 May 2023 17:10:45 GMT
• Title: Context-Aware Transformer Pre-Training for Answer Sentence Selection
• Authors: Luca Di Liello, Siddhant Garg, Alessandro Moschitti
• Abstract summary: We propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. Our experiments show that our pre-training approaches can improve baseline contextual AS2 accuracy by up to 8% on some datasets.
• Score: 102.7383811376319
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answer a given question. The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence. In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. This allows for specializing LMs when fine-tuning for contextual AS2. Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.

Related papers

• Datasets for Multilingual Answer Sentence Selection [59.28492975191415]
  We introduce new high-quality datasets for AS2 in five European languages (French, German, Italian, Portuguese, and Spanish) Results indicate that our datasets are pivotal in producing robust and powerful multilingual AS2 models.
  arXiv  Detail & Related papers  (2024-06-14T16:50:29Z)
• Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection [99.59693674455582]
  We propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics within and across documents. Our experiments on three public and one industrial AS2 datasets demonstrate the empirical superiority of our pre-trained transformers over baseline models.
  arXiv  Detail & Related papers  (2022-05-20T22:39:00Z)
• In Situ Answer Sentence Selection at Web-scale [120.69820139008138]
  Passage-based Extracting Answer Sentence In-place (PEASI) is a novel design for AS2 optimized for Web-scale setting. We train PEASI in a multi-task learning framework that encourages feature sharing between the components: passage reranker and passage-based answer sentence extractor. Experiments show PEASI effectively outperforms the current state-of-the-art setting for AS2, i.e., a point-wise model for ranking sentences independently, by 6.51% in accuracy.
  arXiv  Detail & Related papers  (2022-01-16T06:36:00Z)
• Answer Generation for Retrieval-based Question Answering Systems [80.28727681633096]
  We train a sequence to sequence transformer model to generate an answer from a candidate set. Our tests on three English AS2 datasets show improvement up to 32 absolute points in accuracy over the state of the art.
  arXiv  Detail & Related papers  (2021-06-02T05:45:49Z)
• Modeling Context in Answer Sentence Selection Systems on a Latency Budget [87.45819843513598]
  We present an approach to efficiently incorporate contextual information in AS2 models. For each answer candidate, we first use unsupervised similarity techniques to extract relevant sentences from its source document. Our best approach, which leverages a multi-way attention architecture to efficiently encode context, improves 6% to 11% over nonanswer state of the art in AS2 with minimal impact on system latency.
  arXiv  Detail & Related papers  (2021-01-28T16:24:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.