From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution

• URL: http://arxiv.org/abs/2404.06647v2
• Date: Thu, 11 Apr 2024 02:09:23 GMT
• Title: From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution
• Authors: Bernard J. Koch, David Peterson,
• Abstract summary: Our three-part history of AI research traces the creation of this "epistemic monoculture" back to a radical reconceptualization of scientific progress. We explain how AI's monoculture offers a challenge to the belief that basic, exploration-driven research is needed for scientific progress.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Over the past decade, AI research has focused heavily on building ever-larger deep learning models. This approach has simultaneously unlocked incredible achievements in science and technology, and hindered AI from overcoming long-standing limitations with respect to explainability, ethical harms, and environmental efficiency. Drawing on qualitative interviews and computational analyses, our three-part history of AI research traces the creation of this "epistemic monoculture" back to a radical reconceptualization of scientific progress that began in the late 1980s. In the first era of AI research (1950s-late 1980s), researchers and patrons approached AI as a "basic" science that would advance through autonomous exploration and organic assessments of progress (e.g., peer-review, theoretical consensus). The failure of this approach led to a retrenchment of funding in the 1980s. Amid this "AI Winter," an intervention by the U.S. government reoriented the field towards measurable progress on tasks of military and commercial interest. A new evaluation system called "benchmarking" provided an objective way to quantify progress on tasks by focusing exclusively on increasing predictive accuracy on example datasets. Distilling science down to verifiable metrics clarified the roles of scientists, allowed the field to rapidly integrate talent, and provided clear signals of significance and progress. But history has also revealed a tradeoff to this streamlined approach to science: the consolidation around external interests and inherent conservatism of benchmarking has disincentivized exploration beyond scaling monoculture. In the discussion, we explain how AI's monoculture offers a compelling challenge to the belief that basic, exploration-driven research is needed for scientific progress. Implications for the spread of AI monoculture to other sciences in the era of generative AI are also discussed.

Related papers

• AI Scientists Fail Without Strong Implementation Capability [33.232300349142285]
  The emergence of Artificial Intelligence (AI) Scientist represents a paradigm shift in scientific discovery.<n>Recent AI Scientist studies demonstrate sufficient capabilities for independent scientific discovery.<n>Despite this substantial progress, AI Scientist has yet to produce a groundbreaking achievement in the domain of computer science.
  arXiv  Detail & Related papers  (2025-06-02T06:59:10Z)
• Open and Sustainable AI: challenges, opportunities and the road ahead in the life sciences [50.9036832382286]
  We review the increased erosion of trust in AI research outputs, driven by the issues of poor reusability.<n>We discuss the fragmented components of the AI ecosystem and lack of guiding pathways to best support Open and Sustainable AI.<n>Our work connects researchers with relevant AI resources, facilitating the implementation of sustainable, reusable and transparent AI.
  arXiv  Detail & Related papers  (2025-05-22T12:52:34Z)
• AI-Driven Automation Can Become the Foundation of Next-Era Science of Science Research [58.944125758758936]
  The Science of Science (SoS) explores the mechanisms underlying scientific discovery.<n>The advent of artificial intelligence (AI) presents a transformative opportunity for the next generation of SoS.<n>We outline the advantages of AI over traditional methods, discuss potential limitations, and propose pathways to overcome them.
  arXiv  Detail & Related papers  (2025-05-17T15:01:33Z)
• The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search [16.93028430619359]
  The AI Scientist-v2 is an end-to-end agentic system capable of producing the first entirely AI generated peer-review-accepted workshop paper. It iteratively formulates scientific hypotheses, designs and executes experiments, analyzes and visualizes data, and autonomously authors scientific manuscripts. One manuscript achieved high enough scores to exceed the average human acceptance threshold, marking the first instance of a fully AI-generated paper successfully navigating a peer review.
  arXiv  Detail & Related papers  (2025-04-10T18:44:41Z)
• Scaling Laws in Scientific Discovery with AI and Robot Scientists [72.3420699173245]
  An autonomous generalist scientist (AGS) concept combines agentic AI and embodied robotics to automate the entire research lifecycle. AGS aims to significantly reduce the time and resources needed for scientific discovery. As these autonomous systems become increasingly integrated into the research process, we hypothesize that scientific discovery might adhere to new scaling laws.
  arXiv  Detail & Related papers  (2025-03-28T14:00:27Z)
• Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation [58.064940977804596]
  A plethora of new AI models and tools has been proposed, promising to empower researchers and academics worldwide to conduct their research more effectively and efficiently. Ethical concerns regarding shortcomings of these tools and potential for misuse take a particularly prominent place in our discussion.
  arXiv  Detail & Related papers  (2025-02-07T18:26:45Z)
• Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges [11.232704182001253]
  This paper examines the current state of AI for scientific discovery, highlighting recent progress in large language models and other AI techniques applied to scientific tasks. We then outline key challenges and promising research directions toward developing more comprehensive AI systems for scientific discovery.
  arXiv  Detail & Related papers  (2024-12-16T03:52:20Z)
• Navigating AI in Social Work and Beyond: A Multidisciplinary Review [0.0]
  This review aims to provide a comprehensive yet accessible overview, situating AI within broader societal and academic conversations as 2025 dawns. It briefly analyses AI's real-world impacts, ethical challenges, and implications for social work. It presents a vision for AI-facilitated simulations that could transform social work education through Advanced Personalised Simulation Training.
  arXiv  Detail & Related papers  (2024-10-25T05:51:23Z)
• O1 Replication Journey: A Strategic Progress Report -- Part 1 [52.062216849476776]
  This paper introduces a pioneering approach to artificial intelligence research, embodied in our O1 Replication Journey. Our methodology addresses critical challenges in modern AI research, including the insularity of prolonged team-based projects. We propose the journey learning paradigm, which encourages models to learn not just shortcuts, but the complete exploration process.
  arXiv  Detail & Related papers  (2024-10-08T15:13:01Z)
• Now, Later, and Lasting: Ten Priorities for AI Research, Policy, and Practice [63.20307830884542]
  Next several decades may well be a turning point for humanity, comparable to the industrial revolution. Launched a decade ago, the project is committed to a perpetual series of studies by multidisciplinary experts. We offer ten recommendations for action that collectively address both the short- and long-term potential impacts of AI technologies.
  arXiv  Detail & Related papers  (2024-04-06T22:18:31Z)
• From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape [5.852005817069381]
  The study critically examined the current state and future trajectory of generative Artificial Intelligence (AI) It explored how innovations like Google's Gemini and the anticipated OpenAI Q* project are reshaping research priorities and applications across various domains. The study highlighted the importance of incorporating ethical and human-centric methods in AI development, ensuring alignment with societal norms and welfare.
  arXiv  Detail & Related papers  (2023-12-18T01:11:39Z)
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems [268.585904751315]
  New area of research known as AI for science (AI4Science) Areas aim at understanding the physical world from subatomic (wavefunctions and electron density), atomic (molecules, proteins, materials, and interactions), to macro (fluids, climate, and subsurface) scales. Key common challenge is how to capture physics first principles, especially symmetries, in natural systems by deep learning methods.
  arXiv  Detail & Related papers  (2023-07-17T12:14:14Z)
• The Future of Fundamental Science Led by Generative Closed-Loop Artificial Intelligence [67.70415658080121]
  Recent advances in machine learning and AI are disrupting technological innovation, product development, and society as a whole. AI has contributed less to fundamental science in part because large data sets of high-quality data for scientific practice and model discovery are more difficult to access. Here we explore and investigate aspects of an AI-driven, automated, closed-loop approach to scientific discovery.
  arXiv  Detail & Related papers  (2023-07-09T21:16:56Z)
• Artificial intelligence adoption in the physical sciences, natural sciences, life sciences, social sciences and the arts and humanities: A bibliometric analysis of research publications from 1960-2021 [73.06361680847708]
  In 1960 14% of 333 research fields were related to AI, but this increased to over half of all research fields by 1972, over 80% by 1986 and over 98% in current times. In 1960 14% of 333 research fields were related to AI (many in computer science), but this increased to over half of all research fields by 1972, over 80% by 1986 and over 98% in current times. We conclude that the context of the current surge appears different, and that interdisciplinary AI application is likely to be sustained.
  arXiv  Detail & Related papers  (2023-06-15T14:08:07Z)
• Quantifying the Benefit of Artificial Intelligence for Scientific Research [2.4700789675440524]
  We estimate both the direct use of AI and the potential benefit of AI in scientific research. We find that the use of AI in research is widespread throughout the sciences, growing especially rapidly since 2015. Our analysis reveals considerable potential for AI to benefit numerous scientific fields, yet a notable disconnect exists between AI education and its research applications.
  arXiv  Detail & Related papers  (2023-04-17T08:08:50Z)
• On the Evolution of A.I. and Machine Learning: Towards a Meta-level Measuring and Understanding Impact, Influence, and Leadership at Premier A.I. Conferences [0.26999000177990923]
  We present measures allowing the analyses of AI and machine learning researchers' impact, influence, and leadership over the last decades. We look at papers published at the flagship AI and machine learning conferences since the first International Joint Conference on Artificial Intelligence (IJCAI) held in 1969.
  arXiv  Detail & Related papers  (2022-05-26T03:41:12Z)
• Metaethical Perspectives on 'Benchmarking' AI Ethics [81.65697003067841]
  Benchmarks are seen as the cornerstone for measuring technical progress in Artificial Intelligence (AI) research. An increasingly prominent research area in AI is ethics, which currently has no set of benchmarks nor commonly accepted way for measuring the 'ethicality' of an AI system. We argue that it makes more sense to talk about 'values' rather than 'ethics' when considering the possible actions of present and future AI systems.
  arXiv  Detail & Related papers  (2022-04-11T14:36:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.