Pretrained Transformers for Text Ranking: BERT and Beyond

• URL: http://arxiv.org/abs/2010.06467v3
• Date: Thu, 19 Aug 2021 16:37:02 GMT
• Title: Pretrained Transformers for Text Ranking: BERT and Beyond
• Authors: Jimmy Lin, Rodrigo Nogueira, and Andrew Yates
• Abstract summary: This survey provides an overview of text ranking with neural network architectures known as transformers. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing.
• Score: 53.83210899683987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query. Although the most common formulation of text ranking is search, instances of the task can also be found in many natural language processing applications. This survey provides an overview of text ranking with neural network architectures known as transformers, of which BERT is the best-known example. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing (NLP), information retrieval (IR), and beyond. In this survey, we provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply transformers to text ranking problems and researchers who wish to pursue work in this area. We cover a wide range of modern techniques, grouped into two high-level categories: transformer models that perform reranking in multi-stage architectures and dense retrieval techniques that perform ranking directly. There are two themes that pervade our survey: techniques for handling long documents, beyond typical sentence-by-sentence processing in NLP, and techniques for addressing the tradeoff between effectiveness (i.e., result quality) and efficiency (e.g., query latency, model and index size). Although transformer architectures and pretraining techniques are recent innovations, many aspects of how they are applied to text ranking are relatively well understood and represent mature techniques. However, there remain many open research questions, and thus in addition to laying out the foundations of pretrained transformers for text ranking, this survey also attempts to prognosticate where the field is heading.

Related papers

• Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges [4.588192657854766]
  This survey focuses on approaches that apply pretrained transformer encoders like BERT to information retrieval (IR) We group them into six high-level categories: (i) handling long documents, (ii) integrating semantic information, (iii) balancing effectiveness and efficiency, (iv) predicting the weights of terms, (v) query expansion, and (vi) document expansion. We find that for specific tasks, finely tuned BERT encoders still outperform, and at a lower deployment cost.
  arXiv  Detail & Related papers  (2024-02-18T23:22:40Z)
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks [60.38369406877899]
  Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. Our survey encompasses the identification of the top five application domains for transformer-based models.
  arXiv  Detail & Related papers  (2023-06-11T23:13:51Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets [2.9443230571766854]
  Short text classification is a crucial and challenging aspect of Natural Language Processing. In recent short text research, State of the Art (SOTA) methods for traditional text classification have been unexploited. Our experiments unambiguously demonstrate that Transformers achieve SOTA accuracy on short text classification tasks.
  arXiv  Detail & Related papers  (2022-11-30T10:25:24Z)
• Composable Text Controls in Latent Space with ODEs [97.12426987887021]
  This paper proposes a new efficient approach for composable text operations in the compact latent space of text. By connecting pretrained LMs to the latent space through efficient adaption, we then decode the sampled vectors into desired text sequences. Experiments show that composing those operators within our approach manages to generate or edit high-quality text.
  arXiv  Detail & Related papers  (2022-08-01T06:51:45Z)
• HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking [16.592276887533714]
  Hybrid List Aware Transformer Reranking (HLATR) is a subsequent reranking module to incorporate both retrieval and reranking stage features. HLATR is lightweight and can be easily parallelized with existing text retrieval systems. Empirical experiments on two large-scale text retrieval datasets show that HLATR can efficiently improve the ranking performance of existing multi-stage text retrieval methods.
  arXiv  Detail & Related papers  (2022-05-21T11:38:33Z)
• Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization [18.791701342934605]
  The Query Focused Text Summarization (QFTS) task aims at building systems that generate the summary of the text document(s) based on a given query. A key challenge in addressing this task is the lack of large labeled data for training the summarization model. We address this challenge by exploring a series of domain adaptation techniques.
  arXiv  Detail & Related papers  (2021-12-22T05:34:56Z)
• Transformer Based Language Models for Similar Text Retrieval and Ranking [0.0]
  We introduce novel approaches for effectively applying neural transformer models to similar text retrieval and ranking. By eliminating the bag-of-words-based step, our approach is able to accurately retrieve and rank results even when they have no non-stopwords in common with the query.
  arXiv  Detail & Related papers  (2020-05-10T06:12:53Z)
• Pre-training Tasks for Embedding-based Large-scale Retrieval [68.01167604281578]
  We consider the large-scale query-document retrieval problem. Given a query (e.g., a question), return the set of relevant documents from a large document corpus. We show that the key ingredient of learning a strong embedding-based Transformer model is the set of pre-training tasks.
  arXiv  Detail & Related papers  (2020-02-10T16:44:00Z)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [64.22926988297685]
  Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP) In this paper, we explore the landscape of introducing transfer learning techniques for NLP by a unified framework that converts all text-based language problems into a text-to-text format.
  arXiv  Detail & Related papers  (2019-10-23T17:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.