Zeroth-Order SciML: Non-intrusive Integration of Scientific Software with Deep Learning

• URL: http://arxiv.org/abs/2206.02785v1
• Date: Sat, 4 Jun 2022 17:52:42 GMT
• Title: Zeroth-Order SciML: Non-intrusive Integration of Scientific Software with Deep Learning
• Authors: Ioannis Tsaknakis, Bhavya Kailkhura, Sijia Liu, Donald Loveland, James Diffenderfer, Anna Maria Hiszpanski, Mingyi Hong
• Abstract summary: We propose to integrate the abundance of scientific knowledge sources (SKS) with the deep learning (DL) training process. Existing knowledge integration approaches are limited to using differentiable knowledge source to be compatible with first-order DL training paradigm. We show that proposed scheme is able to effectively integrate scientific knowledge with DL training and is able to outperform purely data-driven model for data-limited scientific applications.
• Score: 46.924429562606086
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Using deep learning (DL) to accelerate and/or improve scientific workflows can yield discoveries that are otherwise impossible. Unfortunately, DL models have yielded limited success in complex scientific domains due to large data requirements. In this work, we propose to overcome this issue by integrating the abundance of scientific knowledge sources (SKS) with the DL training process. Existing knowledge integration approaches are limited to using differentiable knowledge source to be compatible with first-order DL training paradigm. In contrast, our proposed approach treats knowledge source as a black-box in turn allowing to integrate virtually any knowledge source. To enable an end-to-end training of SKS-coupled-DL, we propose to use zeroth-order optimization (ZOO) based gradient-free training schemes, which is non-intrusive, i.e., does not require making any changes to the SKS. We evaluate the performance of our ZOO training scheme on two real-world material science applications. We show that proposed scheme is able to effectively integrate scientific knowledge with DL training and is able to outperform purely data-driven model for data-limited scientific applications. We also discuss some limitations of the proposed method and mention potentially worthwhile future directions.

Related papers

• KBAlign: Efficient Self Adaptation on Specific Knowledge Bases [75.78948575957081]
  Large language models (LLMs) usually rely on retrieval-augmented generation to exploit knowledge materials in an instant manner. We propose KBAlign, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently.
  arXiv  Detail & Related papers  (2024-11-22T08:21:03Z)
• Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning [79.46570165281084]
  We propose a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods. MulKI achieves this through four stages, including Eliciting Ideas, Adding New Ideas, Distinguishing Ideas, and Making Connections. Our method demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks.
  arXiv  Detail & Related papers  (2024-11-11T07:36:19Z)
• Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models [79.28821338925947]
  Domain-Class Incremental Learning is a realistic but challenging continual learning scenario. To handle these diverse tasks, pre-trained Vision-Language Models (VLMs) are introduced for their strong generalizability. This incurs a new problem: the knowledge encoded in the pre-trained VLMs may be disturbed when adapting to new tasks, compromising their inherent zero-shot ability. Existing methods tackle it by tuning VLMs with knowledge distillation on extra datasets, which demands heavy overhead. We propose the Distribution-aware Interference-free Knowledge Integration (DIKI) framework, retaining pre-trained knowledge of
  arXiv  Detail & Related papers  (2024-07-07T12:19:37Z)
• Science-Informed Deep Learning (ScIDL) With Applications to Wireless Communications [11.472232944923558]
  This article provides a tutorial on science-informed deep learning (ScIDL) ScIDL aims to integrate existing scientific knowledge with DL techniques to develop more powerful algorithms. We discuss both recent applications of ScIDL and potential future research directions in the field of wireless communications.
  arXiv  Detail & Related papers  (2024-06-29T02:35:39Z)
• Using Gradient to Boost the Generalization Performance of Deep Learning Models for Fluid Dynamics [0.0]
  We present a novel work to increase the generalization capabilities of Deep Learning. Our strategy has shown good results towards a better generalization of DL networks.
  arXiv  Detail & Related papers  (2022-10-09T10:20:09Z)
• EXPANSE: A Deep Continual / Progressive Learning System for Deep Transfer Learning [1.1024591739346294]
  Current DTL techniques suffer from either catastrophic forgetting dilemma or overly biased pre-trained models. We propose a new continual/progressive learning approach for deep transfer learning to tackle these limitations. We offer a new way of training deep learning models inspired by the human education system.
  arXiv  Detail & Related papers  (2022-05-19T03:54:58Z)
• CosSGD: Nonlinear Quantization for Communication-efficient Federated Learning [62.65937719264881]
  Federated learning facilitates learning across clients without transferring local data on these clients to a central server. We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning. Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
  arXiv  Detail & Related papers  (2020-12-15T12:20:28Z)
• Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources [78.72922528736011]
  We propose a novel approach, black-box adversarial reprogramming (BAR), that repurposes a well-trained black-box machine learning model. Using zeroth order optimization and multi-label mapping techniques, BAR can reprogram a black-box ML model solely based on its input-output responses. BAR outperforms state-of-the-art methods and yields comparable performance to the vanilla adversarial reprogramming method.
  arXiv  Detail & Related papers  (2020-07-17T01:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.