Surveys Considered Harmful? Reflecting on the Use of Surveys in AI Research, Development, and Governance

• URL: http://arxiv.org/abs/2408.01458v1
• Date: Fri, 26 Jul 2024 22:10:49 GMT
• Title: Surveys Considered Harmful? Reflecting on the Use of Surveys in AI Research, Development, and Governance
• Authors: Mohammmad Tahaei, Daricia Wilkinson, Alisa Frik, Michael Muller, Ruba Abu-Salma, Lauren Wilcox,
• Abstract summary: We find that public surveys on AI topics are vulnerable to specific Western knowledge, values, and assumptions in their design. We distill provocations and questions for our community to recognize the limitations of surveys for meeting the goals of engagement.
• Score: 21.430179253600308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Calls for engagement with the public in Artificial Intelligence (AI) research, development, and governance are increasing, leading to the use of surveys to capture people's values, perceptions, and experiences related to AI. In this paper, we critically examine the state of human participant surveys associated with these topics. Through both a reflexive analysis of a survey pilot spanning six countries and a systematic literature review of 44 papers featuring public surveys related to AI, we explore prominent perspectives and methodological nuances associated with surveys to date. We find that public surveys on AI topics are vulnerable to specific Western knowledge, values, and assumptions in their design, including in their positioning of ethical concepts and societal values, lack sufficient critical discourse surrounding deployment strategies, and demonstrate inconsistent forms of transparency in their reporting. Based on our findings, we distill provocations and heuristic questions for our community, to recognize the limitations of surveys for meeting the goals of engagement, and to cultivate shared principles to design, deploy, and interpret surveys cautiously and responsibly.

Related papers

• A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures [50.987594546912725]
  Despite a growing corpus of research in AI privacy and explainability, there is little attention on privacy-preserving model explanations. This article presents the first thorough survey about privacy attacks on model explanations and their countermeasures.
  arXiv  Detail & Related papers  (2024-03-31T12:44:48Z)
• Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits [54.648819983899614]
  General purpose AI seems to have lowered the barriers for the public to use AI and harness its power. We introduce PARTICIP-AI, a framework for laypeople to speculate and assess AI use cases and their impacts.
  arXiv  Detail & Related papers  (2024-03-21T19:12:37Z)
• Advancing Explainable Autonomous Vehicle Systems: A Comprehensive Review and Research Roadmap [4.2330023661329355]
  This study presents a review to discuss the complexities associated with explanation generation and presentation. Our roadmap is underpinned by principles of responsible research and innovation. By exploring these research directions, the study aims to guide the development and deployment of explainable AVs.
  arXiv  Detail & Related papers  (2024-03-19T11:43:41Z)
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing [51.524108608250074]
  Black-box machine learning approaches have become a dominant modeling paradigm for knowledge extraction in remote sensing. We perform a systematic review to identify the key trends in the field and shed light on novel explainable AI approaches. We also give a detailed outlook on the challenges and promising research directions.
  arXiv  Detail & Related papers  (2024-02-21T13:19:58Z)
• Exploring Public Opinion on Responsible AI Through The Lens of Cultural Consensus Theory [0.1813006808606333]
  We applied Cultural Consensus Theory to a nationally representative survey dataset on various aspects of AI. Our results offer valuable insights by identifying shared and contrasting views on responsible AI.
  arXiv  Detail & Related papers  (2024-01-06T20:57:35Z)
• Investigating Responsible AI for Scientific Research: An Empirical Study [4.597781832707524]
  The push for Responsible AI (RAI) in such institutions underscores the increasing emphasis on integrating ethical considerations within AI design and development. This paper aims to assess the awareness and preparedness regarding the ethical risks inherent in AI design and development. Our results have revealed certain knowledge gaps concerning ethical, responsible, and inclusive AI, with limitations in awareness of the available AI ethics frameworks.
  arXiv  Detail & Related papers  (2023-12-15T06:40:27Z)
• Responsible AI Considerations in Text Summarization Research: A Review of Current Practices [89.85174013619883]
  We focus on text summarization, a common NLP task largely overlooked by the responsible AI community. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020-2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
  arXiv  Detail & Related papers  (2023-11-18T15:35:36Z)
• The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis [9.379013474854776]
  We investigate the sociotechnical aspects of sentiment analysis (SA) by critically examining 189 peer-reviewed papers on their applications, models, and datasets. By delving into sociological and technological literature on sentiment, we unveil distinct conceptualizations of this term in domains such as finance, government, and medicine. Our study exposes a lack of explicit definitions and frameworks for characterizing sentiment, resulting in potential challenges and biases.
  arXiv  Detail & Related papers  (2023-10-18T20:42:44Z)
• Stakeholder Participation in AI: Beyond "Add Diverse Stakeholders and Stir" [76.44130385507894]
  This paper aims to ground what we dub a 'participatory turn' in AI design by synthesizing existing literature on participation and through empirical analysis of its current practices. Based on our literature synthesis and empirical research, this paper presents a conceptual framework for analyzing participatory approaches to AI design.
  arXiv  Detail & Related papers  (2021-11-01T17:57:04Z)
• Empowering Local Communities Using Artificial Intelligence [70.17085406202368]
  It has become an important topic to explore the impact of AI on society from a people-centered perspective. Previous works in citizen science have identified methods of using AI to engage the public in research. This article discusses the challenges of applying AI in Community Citizen Science.
  arXiv  Detail & Related papers  (2021-10-05T12:51:11Z)
• ESR: Ethics and Society Review of Artificial Intelligence Research [11.4503292891152]
  We have developed the Ethics and Society Review board (ESR), a feedback panel that works with researchers to mitigate negative ethical and societal aspects of AI research. This article describes the ESR as we have designed and run it over its first year across 41 proposals. We analyze aggregate ESR feedback on these proposals, finding that the panel most commonly identifies issues of harms to minority groups, inclusion of diverse stakeholders in the research plan, dual use, and representation in data.
  arXiv  Detail & Related papers  (2021-06-22T03:23:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.