Loss Landscape Analysis for Reliable Quantized ML Models for Scientific Sensing

• URL: http://arxiv.org/abs/2502.08355v1
• Date: Wed, 12 Feb 2025 12:30:49 GMT
• Title: Loss Landscape Analysis for Reliable Quantized ML Models for Scientific Sensing
• Authors: Tommaso Baldi, Javier Campos, Olivia Weng, Caleb Geniesse, Nhan Tran, Ryan Kastner, Alessandro Biondi,
• Abstract summary: We propose a method to perform empirical analysis of the loss landscape of machine learning (ML) models.<n>Our method allows assessing the robustness of ML models to such effects as a function of quantization precision and under different regularization techniques.
• Score: 41.89148096989836
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we propose a method to perform empirical analysis of the loss landscape of machine learning (ML) models. The method is applied to two ML models for scientific sensing, which necessitates quantization to be deployed and are subject to noise and perturbations due to experimental conditions. Our method allows assessing the robustness of ML models to such effects as a function of quantization precision and under different regularization techniques -- two crucial concerns that remained underexplored so far. By investigating the interplay between performance, efficiency, and robustness by means of loss landscape analysis, we both established a strong correlation between gently-shaped landscapes and robustness to input and weight perturbations and observed other intriguing and non-obvious phenomena. Our method allows a systematic exploration of such trade-offs a priori, i.e., without training and testing multiple models, leading to more efficient development workflows. This work also highlights the importance of incorporating robustness into the Pareto optimization of ML models, enabling more reliable and adaptive scientific sensing systems.

Related papers

• Model Hemorrhage and the Robustness Limits of Large Language Models [119.46442117681147]
  Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment. We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes.
  arXiv  Detail & Related papers  (2025-03-31T10:16:03Z)
• Error-controlled non-additive interaction discovery in machine learning models [8.248260569247595]
  We introduce Diamond, a novel method for trustworthy feature interaction discovery. Diamond uniquely integrates the model-X knockoffs framework to control the false discovery rate (FDR) Our empirical evaluations on both simulated and real datasets demonstrate Diamond's utility in enabling more reliable data-driven scientific discoveries.
  arXiv  Detail & Related papers  (2024-08-30T05:13:11Z)
• MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints [50.61346764110482]
  We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
  arXiv  Detail & Related papers  (2024-04-16T02:18:18Z)
• What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation [55.153595212571375]
  Quantization is a technique for improving the memory and computational efficiency of large language models (LLMs) We propose a new perspective on quantization, viewing it as perturbations added to the weights and activations of LLMs. We conduct experiments with various artificial perturbations to explore their impact on LLM performance.
  arXiv  Detail & Related papers  (2024-03-11T03:42:51Z)
• Replication Study: Enhancing Hydrological Modeling with Physics-Guided Machine Learning [0.0]
  Current hydrological modeling methods combine data-driven Machine Learning algorithms and traditional physics-based models. Despite the accuracy of ML in outcome prediction, the integration of scientific knowledge is crucial for reliable predictions. This study introduces a Physics Informed Machine Learning model, which merges the process understanding of conceptual hydrological models with the predictive efficiency of ML algorithms.
  arXiv  Detail & Related papers  (2024-02-21T16:26:59Z)
• On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning [71.44986275228747]
  In-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs) However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration)
  arXiv  Detail & Related papers  (2023-12-21T11:55:10Z)
• Differentiable modeling to unify machine learning and physical models and advance Geosciences [38.92849886903847]
  We outline the concepts, applicability, and significance of differentiable geoscientific modeling (DG) "Differentiable" refers to accurately and efficiently calculating gradients with respect to model variables. Preliminary evidence suggests DG offers better interpretability and causality than Machine Learning.
  arXiv  Detail & Related papers  (2023-01-10T15:24:14Z)
• Physics-Guided Adversarial Machine Learning for Aircraft Systems Simulation [9.978961706999833]
  This work presents a novel approach, physics-guided adversarial machine learning (ML), that improves the confidence over the physics consistency of the model. Empirical evaluation on two aircraft system performance models shows the effectiveness of our adversarial ML approach.
  arXiv  Detail & Related papers  (2022-09-07T19:23:45Z)
• Putting Density Functional Theory to the Test in Machine-Learning-Accelerated Materials Discovery [2.7810723668216575]
  We describe the advances needed in accuracy, efficiency, and approach beyond what is typical in conventional DFT-based machine learning (ML) For DFT to be trusted for a given data point in a high- throughput screen, it must pass a series of tests. For DFT to be trusted for a given data point in a high- throughput screen, it must pass a series of tests.
  arXiv  Detail & Related papers  (2022-05-06T00:34:50Z)
• Learning continuous models for continuous physics [94.42705784823997]
  We develop a test based on numerical analysis theory to validate machine learning models for science and engineering applications. Our results illustrate how principled numerical analysis methods can be coupled with existing ML training/testing methodologies to validate models for science and engineering applications.
  arXiv  Detail & Related papers  (2022-02-17T07:56:46Z)
• Enhancing predictive skills in physically-consistent way: Physics Informed Machine Learning for Hydrological Processes [1.0635248457021496]
  We develop a Physics Informed Machine Learning (PIML) model that combines the process understanding of conceptual hydrological model with predictive abilities of state-of-the-art ML models. We apply the proposed model to predict the monthly time series of the target (streamflow) and intermediate variables (actual evapotranspiration) in the Narmada river basin in India.
  arXiv  Detail & Related papers  (2021-04-22T12:13:42Z)
• Optimization-driven Machine Learning for Intelligent Reflecting Surfaces Assisted Wireless Networks [82.33619654835348]
  Intelligent surface (IRS) has been employed to reshape the wireless channels by controlling individual scattering elements' phase shifts. Due to the large size of scattering elements, the passive beamforming is typically challenged by the high computational complexity. In this article, we focus on machine learning (ML) approaches for performance in IRS-assisted wireless networks.
  arXiv  Detail & Related papers  (2020-08-29T08:39:43Z)
• Real-Time Model Calibration with Deep Reinforcement Learning [4.707841918805165]
  We propose a novel framework for inference of model parameters based on reinforcement learning. The proposed methodology is demonstrated and evaluated on two model-based diagnostics test cases.
  arXiv  Detail & Related papers  (2020-06-07T00:11:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.