On the definition and importance of interpretability in scientific machine learning

• URL: http://arxiv.org/abs/2505.13510v2
• Date: Wed, 28 May 2025 18:55:13 GMT
• Title: On the definition and importance of interpretability in scientific machine learning
• Authors: Conor Rowan, Alireza Doostan,
• Abstract summary: Researchers in the physical sciences seek not just predictive models, but also to uncover the fundamental principles that govern a system of interest.<n>We argue that researchers in equation discovery and symbolic regression tend to conflate the concept of sparsity with interpretability.<n>Our notion of interpretability emphasizes understanding of the mechanism over mathematical sparsity.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Though neural networks trained on large datasets have been successfully used to describe and predict many physical phenomena, there is a sense among scientists that, unlike traditional scientific models comprising simple mathematical expressions, their findings cannot be integrated into the body of scientific knowledge. Critics of machine learning's inability to produce human-understandable relationships have converged on the concept of "interpretability" as its point of departure from more traditional forms of science. As the growing interest in interpretability has shown, researchers in the physical sciences seek not just predictive models, but also to uncover the fundamental principles that govern a system of interest. However, clarity around a definition of interpretability and the precise role that it plays in science is lacking in the literature. In this work, we argue that researchers in equation discovery and symbolic regression tend to conflate the concept of sparsity with interpretability. We review key papers on interpretable machine learning from outside the scientific community and argue that, though the definitions and methods they propose can inform questions of interpretability for scientific machine learning (SciML), they are inadequate for this new purpose. Noting these deficiencies, we propose an operational definition of interpretability for the physical sciences. Our notion of interpretability emphasizes understanding of the mechanism over mathematical sparsity. Innocuous though it may seem, this emphasis on mechanism shows that sparsity is often unnecessary. It also questions the possibility of interpretable scientific discovery when prior knowledge is lacking. We believe a precise and philosophically informed definition of interpretability in SciML will help focus research efforts toward the most significant obstacles to realizing a data-driven scientific future.

Related papers

• A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i [0.0]
  We argue that Mechanistic Interpretability research is a principled approach to understanding models.<n>We show that Explanatory Faithfulness, an assessment of how well an explanation fits a model, is well-defined.
  arXiv  Detail & Related papers  (2025-05-01T19:08:34Z)
• Interpretable Machine Learning in Physics: A Review [10.77934040629518]
  We aim to establish interpretable machine learning as a core research focus in science.<n>We categorize different aspects of interpretability, discuss machine learning models in terms of both interpretability and performance.<n>We highlight recent advances in interpretable machine learning across many subfields of physics.
  arXiv  Detail & Related papers  (2025-03-30T22:44:40Z)
• Scaling Laws in Scientific Discovery with AI and Robot Scientists [72.3420699173245]
  An autonomous generalist scientist (AGS) concept combines agentic AI and embodied robotics to automate the entire research lifecycle.<n>AGS aims to significantly reduce the time and resources needed for scientific discovery.<n>As these autonomous systems become increasingly integrated into the research process, we hypothesize that scientific discovery might adhere to new scaling laws.
  arXiv  Detail & Related papers  (2025-03-28T14:00:27Z)
• "Cause" is Mechanistic Narrative within Scientific Domains: An Ordinary Language Philosophical Critique of "Causal Machine Learning" [2.5782973781085383]
  Causal Learning has emerged as a major theme of research in statistics and machine learning.<n>In this paper we consider recognizing true cause and effect phenomena.
  arXiv  Detail & Related papers  (2025-01-10T10:36:26Z)
• In Defence of Post-hoc Explainability [0.0]
  We introduce Computational Interpretabilism (CI) as a philosophical framework for post-hoc interpretability in scientific AI.<n> Drawing parallels with human expertise, where post-hoc rationalisation coexists with reliable performance, CI establishes that scientific knowledge emerges through structured model interpretation when properly bounded by empirical validation.
  arXiv  Detail & Related papers  (2024-12-23T06:22:03Z)
• LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery [141.39722070734737]
  We propose to enhance the knowledge-driven, abstract reasoning abilities of Large Language Models with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework. We conduct experiments to demonstrate our framework's efficacy in law discovery and molecular design.
  arXiv  Detail & Related papers  (2024-05-16T03:04:10Z)
• A Review of Neuroscience-Inspired Machine Learning [58.72729525961739]
  Bio-plausible credit assignment is compatible with practically any learning condition and is energy-efficient. In this paper, we survey several vital algorithms that model bio-plausible rules of credit assignment in artificial neural networks. We conclude by discussing the future challenges that will need to be addressed in order to make such algorithms more useful in practical applications.
  arXiv  Detail & Related papers  (2024-02-16T18:05:09Z)
• Diverse Explanations From Data-Driven and Domain-Driven Perspectives in the Physical Sciences [4.442043151145212]
  This Perspective explores the sources and implications of diverse explanations in machine learning applications for physical sciences. We examine how different models, explanation methods, levels of feature attribution, and stakeholder needs can result in varying interpretations of ML outputs. Our analysis underscores the importance of considering multiple perspectives when interpreting ML models in scientific contexts.
  arXiv  Detail & Related papers  (2024-02-01T05:28:28Z)
• SciMON: Scientific Inspiration Machines Optimized for Novelty [68.46036589035539]
  We explore and enhance the ability of neural language models to generate novel scientific directions grounded in literature. We take a dramatic departure with a novel setting in which models use as input background contexts. We present SciMON, a modeling framework that uses retrieval of "inspirations" from past scientific papers.
  arXiv  Detail & Related papers  (2023-05-23T17:12:08Z)
• Intrinsic Physical Concepts Discovery with Object-Centric Predictive Models [86.25460882547581]
  We introduce the PHYsical Concepts Inference NEtwork (PHYCINE), a system that infers physical concepts in different abstract levels without supervision. We show that object representations containing the discovered physical concepts variables could help achieve better performance in causal reasoning tasks.
  arXiv  Detail & Related papers  (2023-03-03T11:52:21Z)
• An Informational Space Based Semantic Analysis for Scientific Texts [62.997667081978825]
  This paper introduces computational methods for semantic analysis and the quantifying the meaning of short scientific texts. The representation of scientific-specific meaning is standardised by replacing the situation representations, rather than psychological properties. The research in this paper conducts the base for the geometric representation of the meaning of texts.
  arXiv  Detail & Related papers  (2022-05-31T11:19:32Z)
• Interpretable and Explainable Machine Learning for Materials Science and Chemistry [2.2175470459999636]
  We summarize applications of interpretability and explainability techniques for materials science and chemistry. We discuss various challenges for interpretable machine learning in materials science and, more broadly, in scientific settings. We showcase a number of exciting developments in other fields that could benefit interpretability in material science and chemistry problems.
  arXiv  Detail & Related papers  (2021-11-01T15:40:36Z)
• Compositional Processing Emerges in Neural Networks Solving Math Problems [100.80518350845668]
  Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed. Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
  arXiv  Detail & Related papers  (2021-05-19T07:24:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.