AI Competitions and Benchmarks: The life cycle of challenges and benchmarks

• URL: http://arxiv.org/abs/2312.05296v1
• Date: Fri, 8 Dec 2023 18:44:10 GMT
• Title: AI Competitions and Benchmarks: The life cycle of challenges and benchmarks
• Authors: Gustavo Stolovitzky, Julio Saez-Rodriguez, Julie Bletz, Jacob Albrecht, Gaia Andreoletti, James C. Costello, Paul Boutros
• Abstract summary: We argue for the need to creatively leverage the scientific research and algorithm development community as an axis of robust innovation. Coordinated community engagement in the analysis of highly complex and massive data has emerged as one approach to find robust methodologies.
• Score: 0.49478969093606673
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Data Science research is undergoing a revolution fueled by the transformative power of technology, the Internet, and an ever increasing computational capacity. The rate at which sophisticated algorithms can be developed is unprecedented, yet they remain outpaced by the massive amounts of data that are increasingly available to researchers. Here we argue for the need to creatively leverage the scientific research and algorithm development community as an axis of robust innovation. Engaging these communities in the scientific discovery enterprise by critical assessments, community experiments, and/or crowdsourcing will multiply opportunities to develop new data driven, reproducible and well benchmarked algorithmic solutions to fundamental and applied problems of current interest. Coordinated community engagement in the analysis of highly complex and massive data has emerged as one approach to find robust methodologies that best address these challenges. When community engagement is done in the form of competitions, also known as challenges, the validation of the analytical methodology is inherently addressed, establishing performance benchmarks. Finally, challenges foster open innovation across multiple disciplines to create communities that collaborate directly or indirectly to address significant scientific gaps. Together, participants can solve important problems as varied as health research, climate change, and social equity. Ultimately, challenges can catalyze and accelerate the synthesis of complex data into knowledge or actionable information, and should be viewed a powerful tool to make lasting social and research contributions.

Related papers

• Academic competitions [61.592427413342975]
  This chapter provides a survey of academic challenges in the context of machine learning and related fields. We review the most influential competitions in the last few years and analyze challenges per area of knowledge. The aims of scientific challenges, their goals, major achievements and expectations for the next few years are reviewed.
  arXiv  Detail & Related papers  (2023-12-01T01:01:04Z)
• Data Science for Social Good [2.8621556092850065]
  We present a framework for "data science for social good" (DSSG) research. We perform an analysis of the literature to empirically demonstrate the paucity of work on DSSG in information systems. We hope that this article and the special issue will spur future DSSG research.
  arXiv  Detail & Related papers  (2023-11-02T15:40:20Z)
• On the Opportunities of Green Computing: A Survey [80.21955522431168]
  Artificial Intelligence (AI) has achieved significant advancements in technology and research with the development over several decades. The needs for high computing power brings higher carbon emission and undermines research fairness. To tackle the challenges of computing resources and environmental impact of AI, Green Computing has become a hot research topic.
  arXiv  Detail & Related papers  (2023-11-01T11:16:41Z)
• The Future of Fundamental Science Led by Generative Closed-Loop Artificial Intelligence [67.70415658080121]
  Recent advances in machine learning and AI are disrupting technological innovation, product development, and society as a whole. AI has contributed less to fundamental science in part because large data sets of high-quality data for scientific practice and model discovery are more difficult to access. Here we explore and investigate aspects of an AI-driven, automated, closed-loop approach to scientific discovery.
  arXiv  Detail & Related papers  (2023-07-09T21:16:56Z)
• A research infrastructure for generating and sharing diversity-aware data [0.0]
  Data flow associated with trend of computerizing aspects of people's diversity in their daily lives is associated with issues concerning people protection and their trust in new technologies. We argue for the development of an end-to-end research infrastructure that enables trustworthy diversity-aware data within a citizen science community.
  arXiv  Detail & Related papers  (2023-06-16T10:43:42Z)
• AI for Science: An Emerging Agenda [30.260160661295682]
  This report documents the programme and the outcomes of Dagstuhl Seminar 22382 "Machine Learning for Science: Bridging Data-Driven and Mechanistic Modelling" The transformative potential of AI stems from its widespread applicability across disciplines, and will only be achieved through integration across research domains. Alongside technical advances, the next wave of progress in the field will come from building a community of machine learning researchers, domain experts, citizen scientists, and engineers.
  arXiv  Detail & Related papers  (2023-03-07T20:21:43Z)
• Coordinated Science Laboratory 70th Anniversary Symposium: The Future of Computing [80.72844751804166]
  In 2021, the Coordinated Science Laboratory CSL hosted the Future of Computing Symposium to celebrate its 70th anniversary. We summarize the major technological points, insights, and directions that speakers brought forward during the symposium. Participants discussed topics related to new computing paradigms, technologies, algorithms, behaviors, and research challenges to be expected in the future.
  arXiv  Detail & Related papers  (2022-10-04T17:32:27Z)
• Research Trends and Applications of Data Augmentation Algorithms [77.34726150561087]
  We identify the main areas of application of data augmentation algorithms, the types of algorithms used, significant research trends, their progression over time and research gaps in data augmentation literature. We expect readers to understand the potential of data augmentation, as well as identify future research directions and open questions within data augmentation research.
  arXiv  Detail & Related papers  (2022-07-18T11:38:32Z)
• Empowering Local Communities Using Artificial Intelligence [70.17085406202368]
  It has become an important topic to explore the impact of AI on society from a people-centered perspective. Previous works in citizen science have identified methods of using AI to engage the public in research. This article discusses the challenges of applying AI in Community Citizen Science.
  arXiv  Detail & Related papers  (2021-10-05T12:51:11Z)
• Big Data Analytics Applying the Fusion Approach of Multicriteria Decision Making with Deep Learning Algorithms [0.0]
  Multicriteria based decision making is one of the key issues to solve for various issues related to the alternative effects in big data analysis. It tends to find a solution based on the latest machine learning techniques that include algorithms like decision making and deep learning mechanism based on multicriteria. In essence, several fields, including business, agriculture, information technology, and computer science, use deep learning and multicriteria-based decision-making problems.
  arXiv  Detail & Related papers  (2021-02-02T05:56:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.