Overview of the TREC 2019 Fair Ranking Track

• URL: http://arxiv.org/abs/2003.11650v1
• Date: Wed, 25 Mar 2020 21:34:58 GMT
• Title: Overview of the TREC 2019 Fair Ranking Track
• Authors: Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
• Abstract summary: The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different content providers. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process.
• Score: 65.15263872493799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different content providers in addition to classic notions of relevance. As part of the benchmark, we defined standardized fairness metrics with evaluation protocols and released a dataset for the fair ranking problem. The 2019 task focused on reranking academic paper abstracts given a query. The objective was to fairly represent relevant authors from several groups that were unknown at the system submission time. Thus, the track emphasized the development of systems which have robust performance across a variety of group definitions. Participants were provided with querylog data (queries, documents, and relevance) from Semantic Scholar. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.

Related papers

• Understanding the User: An Intent-Based Ranking Dataset [2.6145315573431214]
  This paper proposes an approach to augmenting such datasets to annotate informative query descriptions. Our methodology involves utilizing state-of-the-art LLMs to analyze and comprehend the implicit intent within individual queries. By extracting key semantic elements, we construct detailed and contextually rich descriptions for these queries.
  arXiv  Detail & Related papers  (2024-08-30T08:40:59Z)
• Measuring Bias in a Ranked List using Term-based Representations [50.69722973236967]
  We propose a novel metric called TExFAIR (term exposure-based fairness) TExFAIR measures fairness based on the term-based representation of groups in a ranked list. Our experiments show that there is no strong correlation between TExFAIR and NFaiRR, which indicates that TExFAIR measures a different dimension of fairness than NFaiRR.
  arXiv  Detail & Related papers  (2024-03-09T18:24:58Z)
• Revisiting Long-tailed Image Classification: Survey and Benchmarks with New Evaluation Metrics [88.39382177059747]
  A corpus of metrics is designed for measuring the accuracy, robustness, and bounds of algorithms for learning with long-tailed distribution. Based on our benchmarks, we re-evaluate the performance of existing methods on CIFAR10 and CIFAR100 datasets.
  arXiv  Detail & Related papers  (2023-02-03T02:40:54Z)
• Mirror Matching: Document Matching Approach in Seed-driven Document Ranking for Medical Systematic Reviews [31.3220495275256]
  Document ranking is an approach for assisting researchers by providing document rankings where relevant documents are ranked higher than irrelevant ones. We propose a document matching measure named Mirror Matching, which calculates matching scores between medical abstract texts by incorporating common writing patterns.
  arXiv  Detail & Related papers  (2021-12-28T22:27:52Z)
• Overview of the TREC 2020 Fair Ranking Track [64.16623297717642]
  This paper provides an overview of the NIST TREC 2020 Fair Ranking track. The central goal of the Fair Ranking track is to provide fair exposure to different groups of authors.
  arXiv  Detail & Related papers  (2021-08-11T10:22:05Z)
• Corpora Evaluation and System Bias Detection in Multi-document Summarization [25.131744693121508]
  Multi-document summarization (MDS) is the task of reflecting key points from any set of documents into a concise text paragraph. Owing to no standard definition of the task, we encounter a plethora of datasets with varying levels of overlap and conflict between participating documents. New systems report results on a set of chosen datasets, which might not correlate with their performance on the other datasets.
  arXiv  Detail & Related papers  (2020-10-05T05:25:43Z)
• Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary [65.37544133256499]
  We propose a metric to evaluate the content quality of a summary using question-answering (QA) We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval.
  arXiv  Detail & Related papers  (2020-10-01T15:33:09Z)
• Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering [9.480648914353035]
  We present FiRA: a novel dataset of Fine-Grained Relevances. We extend the ranked retrieval annotations of the Deep Learning track of TREC 2019 with passage and word level graded relevance annotations for all relevant documents. As an example, we evaluate the recently introduced TKL document ranking model. We find that although TKL exhibits state-of-the-art retrieval results for long documents, it misses many relevant passages.
  arXiv  Detail & Related papers  (2020-08-12T14:59:50Z)
• Recognizing Families In the Wild: White Paper for the 4th Edition Data Challenge [91.55319616114943]
  This paper summarizes the supported tasks (i.e., kinship verification, tri-subject verification, and search & retrieval of missing children) in the Recognizing Families In the Wild (RFIW) evaluation. The purpose of this paper is to describe the 2020 RFIW challenge, end-to-end, along with forecasts in promising future directions.
  arXiv  Detail & Related papers  (2020-02-15T02:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.