Data and its (dis)contents: A survey of dataset development and use in machine learning research

• URL: http://arxiv.org/abs/2012.05345v1
• Date: Wed, 9 Dec 2020 22:13:13 GMT
• Title: Data and its (dis)contents: A survey of dataset development and use in machine learning research
• Authors: Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, Alex Hanna
• Abstract summary: We survey the many concerns raised about the way we collect and use data in machine learning. We advocate that a more cautious and thorough understanding of data is necessary to address several of the practical and ethical issues of the field.
• Score: 11.042648980854487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Datasets have played a foundational role in the advancement of machine learning research. They form the basis for the models we design and deploy, as well as our primary medium for benchmarking and evaluation. Furthermore, the ways in which we collect, construct and share these datasets inform the kinds of problems the field pursues and the methods explored in algorithm development. However, recent work from a breadth of perspectives has revealed the limitations of predominant practices in dataset collection and use. In this paper, we survey the many concerns raised about the way we collect and use data in machine learning and advocate that a more cautious and thorough understanding of data is necessary to address several of the practical and ethical issues of the field.

Related papers

• Data-Centric AI in the Age of Large Language Models [51.20451986068925]
  This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs) We make the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs. We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization.
  arXiv  Detail & Related papers  (2024-06-20T16:34:07Z)
• Machine Learning Data Practices through a Data Curation Lens: An Evaluation Framework [1.5993707490601146]
  We evaluate data practices in machine learning as data curation practices. We find that researchers in machine learning, which often emphasizes model development, struggle to apply standard data curation principles.
  arXiv  Detail & Related papers  (2024-05-04T16:21:05Z)
• AI Competitions and Benchmarks: Dataset Development [42.164845505628506]
  This chapter provides a comprehensive overview of established methodological tools, enriched by our practical experience. We develop the tasks involved in dataset development and offer insights into their effective management. Then, we provide more details about the implementation process which includes data collection, transformation, and quality evaluation.
  arXiv  Detail & Related papers  (2024-04-15T12:01:42Z)
• From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models [98.41645229835493]
  Data visualization in the form of charts plays a pivotal role in data analysis, offering critical insights and aiding in informed decision-making. Large foundation models, such as large language models, have revolutionized various natural language processing tasks. This survey paper serves as a comprehensive resource for researchers and practitioners in the fields of natural language processing, computer vision, and data analysis.
  arXiv  Detail & Related papers  (2024-03-18T17:57:09Z)
• Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey [17.19337964440007]
  There is currently a lack of comprehensive review that summarizes and compares the key techniques, metrics, datasets, models, and optimization approaches in this research domain. This survey aims to address this gap by consolidating recent progress in these areas, offering a thorough survey and taxonomy of the datasets, metrics, and methodologies utilized. It identifies strengths, limitations, unexplored territories, and gaps in the existing literature, while providing some insights for future research directions in this vital and rapidly evolving field.
  arXiv  Detail & Related papers  (2024-02-27T23:59:01Z)
• Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective [35.620874971064765]
  This work takes a "data-centric" view, examining how data collection, processing, and analysis contribute to explainable AI (XAI) We categorize existing work into three categories subject to their purposes: interpretations of deep models, influences of training data, and insights of domain knowledge. Specifically, we distill XAI methodologies into data mining operations on training and testing data across modalities.
  arXiv  Detail & Related papers  (2024-01-09T06:27:09Z)
• Capture the Flag: Uncovering Data Insights with Large Language Models [90.47038584812925]
  This study explores the potential of using Large Language Models (LLMs) to automate the discovery of insights in data. We propose a new evaluation methodology based on a "capture the flag" principle, measuring the ability of such models to recognize meaningful and pertinent information (flags) in a dataset.
  arXiv  Detail & Related papers  (2023-12-21T14:20:06Z)
• On Responsible Machine Learning Datasets with Fairness, Privacy, and Regulatory Norms [56.119374302685934]
  There have been severe concerns over the trustworthiness of AI technologies. Machine and deep learning algorithms depend heavily on the data used during their development. We propose a framework to evaluate the datasets through a responsible rubric.
  arXiv  Detail & Related papers  (2023-10-24T14:01:53Z)
• Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook [95.32949323258251]
  Temporal data, notably time series andtemporal-temporal data, are prevalent in real-world applications. Recent advances in large language and other foundational models have spurred increased use in time series andtemporal data mining.
  arXiv  Detail & Related papers  (2023-10-16T09:06:00Z)
• Bringing the People Back In: Contesting Benchmark Machine Learning Datasets [11.00769651520502]
  We outline a research program - a genealogy of machine learning data - for investigating how and why these datasets have been created. We describe the ways in which benchmark datasets in machine learning operate as infrastructure and pose four research questions for these datasets.
  arXiv  Detail & Related papers  (2020-07-14T23:22:13Z)
• DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier [58.979104709647295]
  We bridge the gap between the abundance of available data and lack of relevant data, for the future learning tasks of a trained network. We use the available data, that may be an imbalanced subset of the original training dataset, or a related domain dataset, to retrieve representative samples. We demonstrate that data from a related domain can be leveraged to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2019-12-27T02:05:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.