Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy

• URL: http://arxiv.org/abs/2312.14565v2
• Date: Fri, 16 Aug 2024 15:12:36 GMT
• Title: Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy
• Authors: Johann Laux, Fabian Stephany, Alice Liefgreen,
• Abstract summary: The global surge in AI applications is transforming industries, leading to displacement and complementation of existing jobs, while also giving rise to new employment opportunities. Data annotation, encompassing the labelling of images or annotating of texts by human workers, crucially influences the quality of a dataset directly influences the quality of AI models trained on it. This paper delves into the economics of data annotation, with a specific focus on the impact of task instruction design and monetary incentives on data quality and costs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The global surge in AI applications is transforming industries, leading to displacement and complementation of existing jobs, while also giving rise to new employment opportunities. Data annotation, encompassing the labelling of images or annotating of texts by human workers, crucially influences the quality of a dataset directly influences the quality of AI models trained on it. This paper delves into the economics of data annotation, with a specific focus on the impact of task instruction design (that is, the choice between rules and standards as theorised in law and economics) and monetary incentives on data quality and costs. An experimental study involving 307 data annotators examines six groups with varying task instructions (norms) and monetary incentives. Results reveal that annotators provided with clear rules exhibit higher accuracy rates, outperforming those with vague standards by 14%. Similarly, annotators receiving an additional monetary incentive perform significantly better, with the highest accuracy rate recorded in the group working with both clear rules and incentives (87.5% accuracy). In addition, our results show that rules are perceived as being more helpful by annotators than standards and reduce annotators' difficulty in annotating images. These empirical findings underscore the double benefit of rule-based instructions on both data quality and worker wellbeing. Our research design allows us to reveal that, in our study, rules are more cost-efficient in increasing accuracy than monetary incentives. The paper contributes experimental insights to discussions on the economical, ethical, and legal considerations of AI technologies. Addressing policymakers and practitioners, we emphasise the need for a balanced approach in optimising data annotation processes for efficient and ethical AI development and usage.

Related papers

• Learning from "Silly" Questions Improves Large Language Models, But Only Slightly [46.41255699142185]
  This paper aims to explore some hidden factors: the potential interpretations of its success and a large-scale evaluation of the performance. We use GPT-4 to analyze the successful cases of Ruozhiba questions from the perspective of education, psychology, and cognitive science. Surprisingly, our results indicate that rules can significantly improve model performance in certain tasks, while potentially diminishing performance on others.
  arXiv  Detail & Related papers  (2024-11-21T13:45:40Z)
• FairDgcl: Fairness-aware Recommendation with Dynamic Graph Contrastive Learning [48.38344934125999]
  We study how to implement high-quality data augmentation to improve recommendation fairness. Specifically, we propose FairDgcl, a dynamic graph adversarial contrastive learning framework. We show that FairDgcl can simultaneously generate enhanced representations that possess both fairness and accuracy.
  arXiv  Detail & Related papers  (2024-10-23T04:43:03Z)
• Preliminary Insights on Industry Practices for Addressing Fairness Debt [4.546982900370235]
  This study explores how software professionals identify and address biases in AI systems within the software industry. Our paper presents initial evidence on addressing fairness debt and provides a foundation for developing structured guidelines to manage fairness-related issues in AI systems.
  arXiv  Detail & Related papers  (2024-09-04T04:18:42Z)
• Accuracy of training data and model outputs in Generative AI: CREATe Response to the Information Commissioner Office Consultation [6.699484354380871]
  CREATe welcomes the ICO call for evidence on the accuracy of Generative AI. We are happy to highlight aspects of data protection law and AI regulation that we believe should receive attention.
  arXiv  Detail & Related papers  (2024-05-30T10:34:49Z)
• Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media [4.322339935902437]
  Regulatory bodies worldwide are intensifying their efforts to ensure transparency in influencer marketing on social media. The task of automatically detecting sponsored content aims to enable the monitoring and enforcement of such regulations at scale. We propose using chatGPT to augment the annotation process with phrases identified as relevant features and brief explanations.
  arXiv  Detail & Related papers  (2023-06-08T11:29:58Z)
• Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models [73.24381010980606]
  This study examines issues of algorithmic fairness in the context of systems that inform tax audit selection by the IRS. We show how the use of more flexible machine learning methods for selecting audits may affect vertical equity. Our results have implications for the design of algorithmic tools across the public sector.
  arXiv  Detail & Related papers  (2022-06-20T16:27:06Z)
• Whose AI Dream? In search of the aspiration in data annotation [12.454034525520497]
  This paper investigates the work practices concerning data annotation as performed in the industry, in India. Previous investigations have largely focused on annotator subjectivity, bias and efficiency. Our results show that the work of annotators is dictated by the interests, priorities and values of others above their station.
  arXiv  Detail & Related papers  (2022-03-21T06:28:54Z)
• Debiased Pseudo Labeling in Self-Training [77.83549261035277]
  Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. To mitigate the requirement for labeled data, self-training is widely used in both academia and industry by pseudo labeling on readily-available unlabeled data. We propose Debiased, in which the generation and utilization of pseudo labels are decoupled by two independent heads.
  arXiv  Detail & Related papers  (2022-02-15T02:14:33Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)
• Trustworthy AI [75.99046162669997]
  Brittleness to minor adversarial changes in the input data, ability to explain the decisions, address the bias in their training data, are some of the most prominent limitations. We propose the tutorial on Trustworthy AI to address six critical issues in enhancing user and public trust in AI systems.
  arXiv  Detail & Related papers  (2020-11-02T20:04:18Z)
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies [119.07163415116686]
  We train social planners that discover tax policies that can effectively trade-off economic equality and productivity. We present an economic simulation environment that features competitive pressures and market dynamics. We show that AI-driven tax policies improve the trade-off between equality and productivity by 16% over baseline policies.
  arXiv  Detail & Related papers  (2020-04-28T06:57:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.