Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

• URL: http://arxiv.org/abs/2005.04543v1
• Date: Sun, 10 May 2020 01:41:56 GMT
• Title: Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
• Authors: Yang Liu, Michael Gordon, Juntao Wang, Michael Bishop, Yiling Chen, Thomas Pfeiffer, Charles Twardy and Domenico Viganola
• Abstract summary: We review approaches used in the behavioral and social sciences and in the DARPA SCORE project. We focus on the role of human forecasting of replication outcomes. We will discuss opportunities and challenges of using these approaches to monitor and improve the credibility of research areas in Computer Science, AI, and ML.
• Score: 6.485452733699873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The last decade saw the emergence of systematic large-scale replication projects in the social and behavioral sciences, (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014, 2018; Collaboration, 2015). These projects were driven by theoretical and conceptual concerns about a high fraction of "false positives" in the scientific publications (Ioannidis, 2005) (and a high prevalence of "questionable research practices" (Simmons, Nelson, and Simonsohn, 2011). Concerns about the credibility of research findings are not unique to the behavioral and social sciences; within Computer Science, Artificial Intelligence (AI) and Machine Learning (ML) are areas of particular concern (Lucic et al., 2018; Freire, Bonnet, and Shasha, 2012; Gundersen and Kjensmo, 2018; Henderson et al., 2018). Given the pioneering role of the behavioral and social sciences in the promotion of novel methodologies to improve the credibility of research, it is a promising approach to analyze the lessons learned from this field and adjust strategies for Computer Science, AI and ML In this paper, we review approaches used in the behavioral and social sciences and in the DARPA SCORE project. We particularly focus on the role of human forecasting of replication outcomes, and how forecasting can leverage the information gained from relatively labor and resource-intensive replications. We will discuss opportunities and challenges of using these approaches to monitor and improve the credibility of research areas in Computer Science, AI, and ML.

Related papers

• Intelligent Computing Social Modeling and Methodological Innovations in Political Science in the Era of Large Language Models [16.293574791587247]
  This paper proposes the "Intelligent Computing Social Modeling" (ICSM) method to address these issues. By simulating the U.S. presidential election, this study empirically demonstrates the operational pathways and methodological advantages of ICSM. The findings suggest that LLMs will drive methodological innovation in political science through integration and improvement rather than direct substitution.
  arXiv  Detail & Related papers  (2024-10-07T06:30:59Z)
• Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions [1.1970409518725493]
  The article highlights the application areas that could have a positive impact on society along with the ethical considerations. It includes responsible development considerations, algorithmic improvements, ethical challenges, and societal implications.
  arXiv  Detail & Related papers  (2024-09-25T14:36:30Z)
• Data Science for Social Good [2.8621556092850065]
  We present a framework for "data science for social good" (DSSG) research. We perform an analysis of the literature to empirically demonstrate the paucity of work on DSSG in information systems. We hope that this article and the special issue will spur future DSSG research.
  arXiv  Detail & Related papers  (2023-11-02T15:40:20Z)
• On the Opportunities of Green Computing: A Survey [80.21955522431168]
  Artificial Intelligence (AI) has achieved significant advancements in technology and research with the development over several decades. The needs for high computing power brings higher carbon emission and undermines research fairness. To tackle the challenges of computing resources and environmental impact of AI, Green Computing has become a hot research topic.
  arXiv  Detail & Related papers  (2023-11-01T11:16:41Z)
• Survey of Social Bias in Vision-Language Models [65.44579542312489]
  Survey aims to provide researchers with a high-level insight into the similarities and differences of social bias studies in pre-trained models across NLP, CV, and VL. The findings and recommendations presented here can benefit the ML community, fostering the development of fairer and non-biased AI models.
  arXiv  Detail & Related papers  (2023-09-24T15:34:56Z)
• The Future of Fundamental Science Led by Generative Closed-Loop Artificial Intelligence [67.70415658080121]
  Recent advances in machine learning and AI are disrupting technological innovation, product development, and society as a whole. AI has contributed less to fundamental science in part because large data sets of high-quality data for scientific practice and model discovery are more difficult to access. Here we explore and investigate aspects of an AI-driven, automated, closed-loop approach to scientific discovery.
  arXiv  Detail & Related papers  (2023-07-09T21:16:56Z)
• Machine Psychology [54.287802134327485]
  We argue that a fruitful direction for research is engaging large language models in behavioral experiments inspired by psychology. We highlight theoretical perspectives, experimental paradigms, and computational analysis techniques that this approach brings to the table. It paves the way for a "machine psychology" for generative artificial intelligence (AI) that goes beyond performance benchmarks.
  arXiv  Detail & Related papers  (2023-03-24T13:24:41Z)
• AI for Science: An Emerging Agenda [30.260160661295682]
  This report documents the programme and the outcomes of Dagstuhl Seminar 22382 "Machine Learning for Science: Bridging Data-Driven and Mechanistic Modelling" The transformative potential of AI stems from its widespread applicability across disciplines, and will only be achieved through integration across research domains. Alongside technical advances, the next wave of progress in the field will come from building a community of machine learning researchers, domain experts, citizen scientists, and engineers.
  arXiv  Detail & Related papers  (2023-03-07T20:21:43Z)
• Coordinated Science Laboratory 70th Anniversary Symposium: The Future of Computing [80.72844751804166]
  In 2021, the Coordinated Science Laboratory CSL hosted the Future of Computing Symposium to celebrate its 70th anniversary. We summarize the major technological points, insights, and directions that speakers brought forward during the symposium. Participants discussed topics related to new computing paradigms, technologies, algorithms, behaviors, and research challenges to be expected in the future.
  arXiv  Detail & Related papers  (2022-10-04T17:32:27Z)
• Automating Ambiguity: Challenges and Pitfalls of Artificial Intelligence [0.0]
  This thesis weaves together seemingly disparate fields of enquiry to examine core scientific and ethical challenges, pitfalls, and problems of AI. The various challenges, problems and pitfalls of these systems are a hot topic of research in various areas, such as critical data/algorithm studies, science and technology studies (STS), embodied and enactive cognitive science, complexity science, Afro-feminism, and the broadly construed emerging field of Fairness, Accountability, and Transparency (FAccT)
  arXiv  Detail & Related papers  (2022-06-08T22:05:07Z)
• Applications of physics-informed scientific machine learning in subsurface science: A survey [64.0476282000118]
  Geosystems are geological formations altered by humans activities such as fossil energy exploration, waste disposal, geologic carbon sequestration, and renewable energy generation. The responsible use and exploration of geosystems are thus critical to the geosystem governance, which in turn depends on the efficient monitoring, risk assessment, and decision support tools for practical implementation. Fast advances in machine learning algorithms and novel sensing technologies in recent years have presented new opportunities for the subsurface research community to improve the efficacy and transparency of geosystem governance.
  arXiv  Detail & Related papers  (2021-04-10T13:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.