Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity

• URL: http://arxiv.org/abs/2207.00751v1
• Date: Sat, 2 Jul 2022 06:28:25 GMT
• Title: Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity
• Authors: Jianyi Yang and Shaolei Ren
• Abstract summary: We study how and why domain knowledge benefits the performance of informed learning. We propose a generalized informed training objective to better exploit the benefits of knowledge and balance the label and knowledge imperfectness.
• Score: 27.84415856657607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By integrating domain knowledge with labeled samples, informed machine learning has been emerging to improve the learning performance for a wide range of applications. Nonetheless, rigorous understanding of the role of injected domain knowledge has been under-explored. In this paper, we consider an informed deep neural network (DNN) with over-parameterization and domain knowledge integrated into its training objective function, and study how and why domain knowledge benefits the performance. Concretely, we quantitatively demonstrate the two benefits of domain knowledge in informed learning - regularizing the label-based supervision and supplementing the labeled samples - and reveal the trade-off between label and knowledge imperfectness in the bound of the population risk. Based on the theoretical analysis, we propose a generalized informed training objective to better exploit the benefits of knowledge and balance the label and knowledge imperfectness, which is validated by the population risk bound. Our analysis on sampling complexity sheds lights on how to choose the hyper-parameters for informed learning, and further justifies the advantages of knowledge informed learning.

Related papers

• Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods [45.1662948487385]
  Language Models (LMs) acquire parametric knowledge from their training process, embedding it within their weights. Instance Attribution (IA) and Neuron Attribution (NA) offer insights into this training-acquired knowledge. Our study introduces a novel evaluation framework to quantify and compare the knowledge revealed by IA and NA.
  arXiv  Detail & Related papers  (2024-04-29T12:38:26Z)
• Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations [0.0]
  Large language models (LLMs) and generative AI have recently opened new doors for generating human-like explanations. This paper proposes an approach to utilize knowledge graphs (KG) as a source of factual context. We utilize the semantic relations in the knowledge graph to offer curated knowledge about learning recommendations.
  arXiv  Detail & Related papers  (2024-03-05T14:41:12Z)
• InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration [61.554209059971576]
  Large Language Models (LLMs) have shown remarkable open-generation capabilities across diverse domains. Injecting new knowledge poses the risk of forgetting previously acquired knowledge. We propose a novel Infuser-Guided Knowledge Integration framework.
  arXiv  Detail & Related papers  (2024-02-18T03:36:26Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators [78.63553017938911]
  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. We introduce CONNER, designed to evaluate generated knowledge from six important perspectives.
  arXiv  Detail & Related papers  (2023-10-11T08:22:37Z)
• Worth of knowledge in deep learning [3.132595571344153]
  We present a framework inspired by interpretable machine learning to evaluate the worth of knowledge. Our findings elucidate the complex relationship between data and knowledge, including dependence, synergistic, and substitution effects. Our model-agnostic framework can be applied to a variety of common network architectures, providing a comprehensive understanding of the role of prior knowledge in deep learning models.
  arXiv  Detail & Related papers  (2023-07-03T02:25:19Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• Knowledge-augmented Deep Learning and Its Applications: A Survey [60.221292040710885]
  knowledge-augmented deep learning (KADL) aims to identify domain knowledge and integrate it into deep models for data-efficient, generalizable, and interpretable deep learning. This survey subsumes existing works and offers a bird's-eye view of research in the general area of knowledge-augmented deep learning.
  arXiv  Detail & Related papers  (2022-11-30T03:44:15Z)
• Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge [77.77684299758494]
  We extensively investigate approaches to enhancing the answer selection model with external knowledge from knowledge graph (KG) First, we present a context-knowledge interaction learning framework, Knowledge-aware Neural Network (KNN), which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information. To handle the diversity and complexity of KG information, we propose a Contextualized Knowledge-aware Attentive Neural Network (CKANN), which improves the knowledge representation learning with structure information via a customized Graph Convolutional Network (GCN) and comprehensively learns context-based and knowledge-based sentence representation via
  arXiv  Detail & Related papers  (2021-04-12T05:52:20Z)
• A Quantitative Perspective on Values of Domain Knowledge for Machine Learning [27.84415856657607]
  Domain knowledge in various forms has been playing a crucial role in improving the learning performance. We study the problem of quantifying the values of domain knowledge in terms of its contribution to the learning performance.
  arXiv  Detail & Related papers  (2020-11-17T06:12:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.