Dead Zone of Accountability: Why Social Claims in Machine Learning Research Should Be Articulated and Defended

• URL: http://arxiv.org/abs/2508.08739v3
• Date: Thu, 14 Aug 2025 21:34:49 GMT
• Title: Dead Zone of Accountability: Why Social Claims in Machine Learning Research Should Be Articulated and Defended
• Authors: Tianqi Kou, Dana Calacci, Cindy Lin,
• Abstract summary: Many Machine Learning research studies use language that describes potential social benefits or technical affordances of new methods and technologies.<n>Such language, which we call "social claims", can help garner substantial resources and influence for those involved in ML research and technology production.<n>This paper investigates the claim-reality gap and makes a normative argument for developing accountability mechanisms for it.
• Score: 1.6385815610837167
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many Machine Learning research studies use language that describes potential social benefits or technical affordances of new methods and technologies. Such language, which we call "social claims", can help garner substantial resources and influence for those involved in ML research and technology production. However, there exists a gap between social claims and reality (the claim-reality gap): ML methods often fail to deliver the claimed functionality or social impacts. This paper investigates the claim-reality gap and makes a normative argument for developing accountability mechanisms for it. In making the argument, we make three contributions. First, we show why the symptom - absence of social claim accountability - is problematic. Second, we coin dead zone of accountability - a lens that scholars and practitioners can use to identify opportunities for new forms of accountability. We apply this lens to the claim-reality gap and provide a diagnosis by identifying cognitive and structural resistances to accountability in the claim-reality gap. Finally, we offer a prescription - two potential collaborative research agendas that can help create the condition for social claim accountability.

Related papers

• Social Simulations with Large Language Model Risk Utopian Illusion [61.358959720048354]
  We introduce a systematic framework for analyzing large language models' behavior in social simulation.<n>Our approach simulates multi-agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions.<n>Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it.
  arXiv  Detail & Related papers  (2025-10-24T06:08:41Z)
• SocialNLI: A Dialogue-Centric Social Inference Dataset [49.60157928163403]
  We introduce SocialNLI -- the first social dialogue inference dataset.<n>SocialNLI consists of a collection of dialogue transcripts hand-picked to center complex social nuances.<n>We evaluate reasoning models theory-of-mind ability through multi-step counterfactual reasoning.
  arXiv  Detail & Related papers  (2025-10-06T23:42:01Z)
• The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models' Social Reasoning [49.32390524168273]
  Our work aims to advance the social reasoning of embodied artificial intelligence (AI) agents in real-world social interactions.<n>We introduce a large-scale real-world Human Robot Social Interaction (HSRI) dataset to benchmark the capabilities of language models (LMs) and foundational models (FMs)<n>Our dataset consists of 400 real-world human social robot interaction videos and over 10K annotations, detailing the robot's social errors, competencies, rationale, and corrective actions.
  arXiv  Detail & Related papers  (2025-04-07T06:27:02Z)
• Social Genome: Grounded Social Reasoning Abilities of Multimodal Models [61.88413918026431]
  Social reasoning abilities are crucial for AI systems to interpret and respond to multimodal human communication and interaction within social contexts.<n>We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models.
  arXiv  Detail & Related papers  (2025-02-21T00:05:40Z)
• Can LLMs Assist Annotators in Identifying Morality Frames? -- Case Study on Vaccination Debate on Social Media [22.976609127865732]
  Large language models (LLMs) are adept at adapting new tasks through few-shot learning.<n>Our research explores LLMs' potential to assist human annotators in identifying morality frames within vaccination debates on social media.
  arXiv  Detail & Related papers  (2025-02-04T04:10:23Z)
• From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap [0.0]
  Two goals - improving replicability and accountability of Machine Learning research. This paper posits that reconceptualizing replicability can help bridge the gap.
  arXiv  Detail & Related papers  (2024-04-19T18:36:14Z)
• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs [77.88043871260466]
  We show that one of today's largest language models lacks this kind of social intelligence out-of-the box. We conclude that person-centric NLP approaches might be more effective towards neural Theory of Mind.
  arXiv  Detail & Related papers  (2022-10-24T14:58:58Z)
• Fair Machine Learning in Healthcare: A Review [90.22219142430146]
  We analyze the intersection of fairness in machine learning and healthcare disparities. We provide a critical review of the associated fairness metrics from a machine learning standpoint. We propose several new research directions that hold promise for developing ethical and equitable ML applications in healthcare.
  arXiv  Detail & Related papers  (2022-06-29T04:32:10Z)
• Machine learning in the social and health sciences [0.8681909776958184]
  This paper provides a meta-mapping of research questions in the social and health sciences to appropriate machine learning approaches. We map the established classification into description, prediction, and causal inference to common research goals.
  arXiv  Detail & Related papers  (2021-06-20T15:48:45Z)
• The Use and Misuse of Counterfactuals in Ethical Machine Learning [2.28438857884398]
  We argue for more caution with the use of counterfactuals when the facts to be considered are social categories such as race or gender. We conclude that the counterfactual approach in machine learning fairness and social explainability can require an incoherent theory of what social categories are.
  arXiv  Detail & Related papers  (2021-02-09T19:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.