The Implications of Open Generative Models in Human-Centered Data Science Work: A Case Study with Fact-Checking Organizations

• URL: http://arxiv.org/abs/2408.01962v1
• Date: Sun, 4 Aug 2024 08:41:48 GMT
• Title: The Implications of Open Generative Models in Human-Centered Data Science Work: A Case Study with Fact-Checking Organizations
• Authors: Robert Wolfe, Tanushree Mitra,
• Abstract summary: We focus on the impact of open models on organizations, which use AI to observe and analyze large volumes of circulating misinformation. We conducted an interview study with N=24 professionals at 20 fact-checking organizations on six continents. We find that fact-checking organizations prefer open models for Organizational Autonomy, Data Privacy and Ownership, Application Specificity, and Capability Transparency.
• Score: 12.845170214324662
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Calls to use open generative language models in academic research have highlighted the need for reproducibility and transparency in scientific research. However, the impact of generative AI extends well beyond academia, as corporations and public interest organizations have begun integrating these models into their data science pipelines. We expand this lens to include the impact of open models on organizations, focusing specifically on fact-checking organizations, which use AI to observe and analyze large volumes of circulating misinformation, yet must also ensure the reproducibility and impartiality of their work. We wanted to understand where fact-checking organizations use open models in their data science pipelines; what motivates their use of open models or proprietary models; and how their use of open or proprietary models can inform research on the societal impact of generative AI. To answer these questions, we conducted an interview study with N=24 professionals at 20 fact-checking organizations on six continents. Based on these interviews, we offer a five-component conceptual model of where fact-checking organizations employ generative AI to support or automate parts of their data science pipeline, including Data Ingestion, Data Analysis, Data Retrieval, Data Delivery, and Data Sharing. We then provide taxonomies of fact-checking organizations' motivations for using open models and the limitations that prevent them for further adopting open models, finding that they prefer open models for Organizational Autonomy, Data Privacy and Ownership, Application Specificity, and Capability Transparency. However, they nonetheless use proprietary models due to perceived advantages in Performance, Usability, and Safety, as well as Opportunity Costs related to participation in emerging generative AI ecosystems. Our work provides novel perspective on open models in data-driven organizations.

Related papers

• Generative AI like ChatGPT in Blockchain Federated Learning: use cases, opportunities and future [4.497001527881303]
  This research explores potential integrations of generative AI in federated learning. generative adversarial networks (GANs) and variational autoencoders (VAEs) Generating synthetic data helps federated learning address challenges related to limited data availability.
  arXiv  Detail & Related papers  (2024-07-25T19:43:49Z)
• The Impact and Opportunities of Generative AI in Fact-Checking [12.845170214324662]
  Generative AI appears poised to transform white collar professions, with more than 90% of Fortune 500 companies using OpenAI's flagship GPT models. But how will such technologies impact organizations whose job is to verify and report factual information? We conducted 30 interviews with N=38 participants working at 29 fact-checking organizations across six continents.
  arXiv  Detail & Related papers  (2024-05-24T23:58:01Z)
• Best Practices and Lessons Learned on Synthetic Data [83.63271573197026]
  The success of AI models relies on the availability of large, diverse, and high-quality datasets. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns.
  arXiv  Detail & Related papers  (2024-04-11T06:34:17Z)
• The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence [0.0]
  We introduce the Model Openness Framework (MOF), a three-tiered ranked classification system that rates machine learning models based on their completeness and openness. For each MOF class, we specify code, data, and documentation components of the model development lifecycle that must be released and under which open licenses. In addition, the Model Openness Tool (MOT) provides a user-friendly reference implementation to evaluate the openness and completeness of models against the MOF classification system.
  arXiv  Detail & Related papers  (2024-03-20T17:47:08Z)
• On Responsible Machine Learning Datasets with Fairness, Privacy, and Regulatory Norms [56.119374302685934]
  There have been severe concerns over the trustworthiness of AI technologies. Machine and deep learning algorithms depend heavily on the data used during their development. We propose a framework to evaluate the datasets through a responsible rubric.
  arXiv  Detail & Related papers  (2023-10-24T14:01:53Z)
• Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment [66.91538273487379]
  There is a certain consensus about the need to develop AI applications with a Human-Centric approach. Human-Centric Machine Learning needs to be developed based on four main requirements: (i) utility and social good; (ii) privacy and data ownership; (iii) transparency and accountability; and (iv) fairness in AI-driven decision-making processes. We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data.
  arXiv  Detail & Related papers  (2023-02-13T16:44:44Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Application of Federated Learning in Building a Robust COVID-19 Chest X-ray Classification Model [0.0]
  Federated Learning (FL) helps AI models to generalize better without moving all the data to a central server. We trained a deep learning model to solve a binary classification problem of predicting the presence or absence of COVID-19.
  arXiv  Detail & Related papers  (2022-04-22T05:21:50Z)
• On the Opportunities and Risks of Foundation Models [256.61956234436553]
  We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration.
  arXiv  Detail & Related papers  (2021-08-16T17:50:08Z)
• Explainable Patterns: Going from Findings to Insights to Support Data Analytics Democratization [60.18814584837969]
  We present Explainable Patterns (ExPatt), a new framework to support lay users in exploring and creating data storytellings. ExPatt automatically generates plausible explanations for observed or selected findings using an external (textual) source of information.
  arXiv  Detail & Related papers  (2021-01-19T16:13:44Z)
• Principles and Practice of Explainable Machine Learning [12.47276164048813]
  This report focuses on data-driven methods -- machine learning (ML) and pattern recognition models in particular. With the increasing prevalence and complexity of methods, business stakeholders in the very least have a growing number of concerns about the drawbacks of models. We have undertaken a survey to help industry practitioners understand the field of explainable machine learning better.
  arXiv  Detail & Related papers  (2020-09-18T14:50:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.