Overview of the TREC 2022 deep learning track

• URL: http://arxiv.org/abs/2507.10865v1
• Date: Thu, 10 Jul 2025 20:48:22 GMT
• Title: Overview of the TREC 2022 deep learning track
• Authors: Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, Ian Soboroff,
• Abstract summary: This is the fourth year of the TREC Deep Learning track.<n>We leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available.<n>Similar to previous years, deep neural ranking models that employ large scale pretraining continued to outperform traditional retrieval methods.
• Score: 67.86242254073656
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This is the fourth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we also leverage both the refreshed passage and document collections that were released last year leading to a nearly $16$ times increase in the size of the passage collection and nearly four times increase in the document collection size. Unlike previous years, in 2022 we mainly focused on constructing a more complete test collection for the passage retrieval task, which has been the primary focus of the track. The document ranking task was kept as a secondary task, where document-level labels were inferred from the passage-level labels. Our analysis shows that similar to previous years, deep neural ranking models that employ large scale pretraining continued to outperform traditional retrieval methods. Due to the focusing our judging resources on passage judging, we are more confident in the quality of this year's queries and judgments, with respect to our ability to distinguish between runs and reuse the dataset in future. We also see some surprises in overall outcomes. Some top-performing runs did not do dense retrieval. Runs that did single-stage dense retrieval were not as competitive this year as they were last year.

Related papers

• Overview of the TREC 2021 deep learning track [68.66107744993546]
  This is the third year of the TREC Deep Learning track.<n>We leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks.<n>Deep neural ranking models that employ large scale pretraininig continued to outperform traditional retrieval methods this year.
  arXiv  Detail & Related papers  (2025-07-10T21:58:41Z)
• Overview of the TREC 2023 deep learning track [67.56975103581688]
  This is the fifth year of the TREC Deep Learning track.<n>We leverage the MS MARCO datasets that made hundreds of thousands of human-annotated training labels available.<n>This year we generated synthetic queries using a fine-tuned T5 model and using a GPT-4 prompt.
  arXiv  Detail & Related papers  (2025-07-10T20:39:42Z)
• DAPR: A Benchmark on Document-Aware Passage Retrieval [57.45793782107218]
  We propose and name this task emphDocument-Aware Passage Retrieval (DAPR) While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context. Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
  arXiv  Detail & Related papers  (2023-05-23T10:39:57Z)
• Fine-Grained Distillation for Long Document Retrieval [86.39802110609062]
  Long document retrieval aims to fetch query-relevant documents from a large-scale collection. Knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. We propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers.
  arXiv  Detail & Related papers  (2022-12-20T17:00:36Z)
• Overview of the TREC 2020 deep learning track [30.531644711518414]
  This year we have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in the large data regime.
  arXiv  Detail & Related papers  (2021-02-15T16:47:00Z)
• Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering [9.480648914353035]
  We present FiRA: a novel dataset of Fine-Grained Relevances. We extend the ranked retrieval annotations of the Deep Learning track of TREC 2019 with passage and word level graded relevance annotations for all relevant documents. As an example, we evaluate the recently introduced TKL document ranking model. We find that although TKL exhibits state-of-the-art retrieval results for long documents, it misses many relevant passages.
  arXiv  Detail & Related papers  (2020-08-12T14:59:50Z)
• Overview of the TREC 2019 deep learning track [36.23357487158591]
  The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks. This year 15 groups submitted a total of 75 runs, using various combinations of deep learning, transfer learning and traditional IR ranking methods.
  arXiv  Detail & Related papers  (2020-03-17T17:12:36Z)
• Pre-training Tasks for Embedding-based Large-scale Retrieval [68.01167604281578]
  We consider the large-scale query-document retrieval problem. Given a query (e.g., a question), return the set of relevant documents from a large document corpus. We show that the key ingredient of learning a strong embedding-based Transformer model is the set of pre-training tasks.
  arXiv  Detail & Related papers  (2020-02-10T16:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.