Loss Landscape Analysis for Reliable Quantized ML Models for Scientific Sensing

• URL: http://arxiv.org/abs/2502.08355v1
• Date: Wed, 12 Feb 2025 12:30:49 GMT
• Title: Loss Landscape Analysis for Reliable Quantized ML Models for Scientific Sensing
• Authors: Tommaso Baldi, Javier Campos, Olivia Weng, Caleb Geniesse, Nhan Tran, Ryan Kastner, Alessandro Biondi,
• Abstract summary: We propose a method to perform empirical analysis of the loss landscape of machine learning (ML) models. Our method allows assessing the robustness of ML models to such effects as a function of quantization precision and under different regularization techniques.
• Score: 41.89148096989836
• License:
• Abstract: In this paper, we propose a method to perform empirical analysis of the loss landscape of machine learning (ML) models. The method is applied to two ML models for scientific sensing, which necessitates quantization to be deployed and are subject to noise and perturbations due to experimental conditions. Our method allows assessing the robustness of ML models to such effects as a function of quantization precision and under different regularization techniques -- two crucial concerns that remained underexplored so far. By investigating the interplay between performance, efficiency, and robustness by means of loss landscape analysis, we both established a strong correlation between gently-shaped landscapes and robustness to input and weight perturbations and observed other intriguing and non-obvious phenomena. Our method allows a systematic exploration of such trade-offs a priori, i.e., without training and testing multiple models, leading to more efficient development workflows. This work also highlights the importance of incorporating robustness into the Pareto optimization of ML models, enabling more reliable and adaptive scientific sensing systems.

Related papers

• See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition [56.87609859444084]
  parameter-efficient fine-tuning (PEFT) focuses on optimizing a select subset of parameters while keeping the rest fixed, significantly lowering computational and storage overheads. We take the first step to unify all approaches by dissecting them from a decomposition perspective. We introduce two novel PEFT methods alongside a simple yet effective framework designed to enhance the performance of PEFT techniques across various applications.
  arXiv  Detail & Related papers  (2024-07-07T15:44:42Z)
• MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints [50.61346764110482]
  We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
  arXiv  Detail & Related papers  (2024-04-16T02:18:18Z)
• What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation [55.153595212571375]
  Quantization is a technique for improving the memory and computational efficiency of large language models (LLMs) We propose a new perspective on quantization, viewing it as perturbations added to the weights and activations of LLMs. We conduct experiments with various artificial perturbations to explore their impact on LLM performance.
  arXiv  Detail & Related papers  (2024-03-11T03:42:51Z)
• Replication Study: Enhancing Hydrological Modeling with Physics-Guided Machine Learning [0.0]
  Current hydrological modeling methods combine data-driven Machine Learning algorithms and traditional physics-based models. Despite the accuracy of ML in outcome prediction, the integration of scientific knowledge is crucial for reliable predictions. This study introduces a Physics Informed Machine Learning model, which merges the process understanding of conceptual hydrological models with the predictive efficiency of ML algorithms.
  arXiv  Detail & Related papers  (2024-02-21T16:26:59Z)
• On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning [71.44986275228747]
  In-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs) However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration)
  arXiv  Detail & Related papers  (2023-12-21T11:55:10Z)
• Physics-Guided Adversarial Machine Learning for Aircraft Systems Simulation [9.978961706999833]
  This work presents a novel approach, physics-guided adversarial machine learning (ML), that improves the confidence over the physics consistency of the model. Empirical evaluation on two aircraft system performance models shows the effectiveness of our adversarial ML approach.
  arXiv  Detail & Related papers  (2022-09-07T19:23:45Z)
• Putting Density Functional Theory to the Test in Machine-Learning-Accelerated Materials Discovery [2.7810723668216575]
  We describe the advances needed in accuracy, efficiency, and approach beyond what is typical in conventional DFT-based machine learning (ML) For DFT to be trusted for a given data point in a high- throughput screen, it must pass a series of tests. For DFT to be trusted for a given data point in a high- throughput screen, it must pass a series of tests.
  arXiv  Detail & Related papers  (2022-05-06T00:34:50Z)
• Enhancing predictive skills in physically-consistent way: Physics Informed Machine Learning for Hydrological Processes [1.0635248457021496]
  We develop a Physics Informed Machine Learning (PIML) model that combines the process understanding of conceptual hydrological model with predictive abilities of state-of-the-art ML models. We apply the proposed model to predict the monthly time series of the target (streamflow) and intermediate variables (actual evapotranspiration) in the Narmada river basin in India.
  arXiv  Detail & Related papers  (2021-04-22T12:13:42Z)
• Optimization-driven Machine Learning for Intelligent Reflecting Surfaces Assisted Wireless Networks [82.33619654835348]
  Intelligent surface (IRS) has been employed to reshape the wireless channels by controlling individual scattering elements' phase shifts. Due to the large size of scattering elements, the passive beamforming is typically challenged by the high computational complexity. In this article, we focus on machine learning (ML) approaches for performance in IRS-assisted wireless networks.
  arXiv  Detail & Related papers  (2020-08-29T08:39:43Z)
• Real-Time Model Calibration with Deep Reinforcement Learning [4.707841918805165]
  We propose a novel framework for inference of model parameters based on reinforcement learning. The proposed methodology is demonstrated and evaluated on two model-based diagnostics test cases.
  arXiv  Detail & Related papers  (2020-06-07T00:11:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.