Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity

• URL: http://arxiv.org/abs/2207.00751v1
• Date: Sat, 2 Jul 2022 06:28:25 GMT
• Title: Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity
• Authors: Jianyi Yang and Shaolei Ren
• Abstract summary: We study how and why domain knowledge benefits the performance of informed learning. We propose a generalized informed training objective to better exploit the benefits of knowledge and balance the label and knowledge imperfectness.
• Score: 27.84415856657607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By integrating domain knowledge with labeled samples, informed machine learning has been emerging to improve the learning performance for a wide range of applications. Nonetheless, rigorous understanding of the role of injected domain knowledge has been under-explored. In this paper, we consider an informed deep neural network (DNN) with over-parameterization and domain knowledge integrated into its training objective function, and study how and why domain knowledge benefits the performance. Concretely, we quantitatively demonstrate the two benefits of domain knowledge in informed learning - regularizing the label-based supervision and supplementing the labeled samples - and reveal the trade-off between label and knowledge imperfectness in the bound of the population risk. Based on the theoretical analysis, we propose a generalized informed training objective to better exploit the benefits of knowledge and balance the label and knowledge imperfectness, which is validated by the population risk bound. Our analysis on sampling complexity sheds lights on how to choose the hyper-parameters for informed learning, and further justifies the advantages of knowledge informed learning.

Related papers

• Reliability Across Parametric and External Knowledge: Understanding Knowledge Handling in LLMs [11.860265967829884]
  Large Language Models (LLMs) enhance their problem-solving capability by leveraging both parametric and external knowledge. We introduce a framework for analyzing knowledge-handling based on two key dimensions: the presence of parametric knowledge and the informativeness of external knowledge. We demonstrate that training on data constructed based on the knowledge-handling scenarios improves LLMs' reliability in integrating and utilizing knowledge.
  arXiv  Detail & Related papers  (2025-02-19T11:49:23Z)
• How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training [92.88889953768455]
  Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge. We identify computational subgraphs that facilitate knowledge storage and processing.
  arXiv  Detail & Related papers  (2025-02-16T16:55:43Z)
• LEKA:LLM-Enhanced Knowledge Augmentation [24.552995956148145]
  Humans excel in analogical learning and knowledge transfer. Models would transition from passively acquiring to actively accessing and learning from knowledge. We develop a knowledge augmentation method LEKA for knowledge transfer.
  arXiv  Detail & Related papers  (2025-01-29T17:44:57Z)
• InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration [58.61492157691623]
  Methods for integrating knowledge have been developed, which augment LLMs with domain-specific knowledge graphs through external modules. Our research focuses on a novel problem: efficiently integrating unknown knowledge into LLMs without unnecessary overlap of known knowledge. A risk of introducing new knowledge is the potential forgetting of existing knowledge.
  arXiv  Detail & Related papers  (2024-02-18T03:36:26Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators [78.63553017938911]
  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. We introduce CONNER, designed to evaluate generated knowledge from six important perspectives.
  arXiv  Detail & Related papers  (2023-10-11T08:22:37Z)
• Worth of knowledge in deep learning [3.132595571344153]
  We present a framework inspired by interpretable machine learning to evaluate the worth of knowledge. Our findings elucidate the complex relationship between data and knowledge, including dependence, synergistic, and substitution effects. Our model-agnostic framework can be applied to a variety of common network architectures, providing a comprehensive understanding of the role of prior knowledge in deep learning models.
  arXiv  Detail & Related papers  (2023-07-03T02:25:19Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge [77.77684299758494]
  We extensively investigate approaches to enhancing the answer selection model with external knowledge from knowledge graph (KG) First, we present a context-knowledge interaction learning framework, Knowledge-aware Neural Network (KNN), which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information. To handle the diversity and complexity of KG information, we propose a Contextualized Knowledge-aware Attentive Neural Network (CKANN), which improves the knowledge representation learning with structure information via a customized Graph Convolutional Network (GCN) and comprehensively learns context-based and knowledge-based sentence representation via
  arXiv  Detail & Related papers  (2021-04-12T05:52:20Z)
• A Quantitative Perspective on Values of Domain Knowledge for Machine Learning [27.84415856657607]
  Domain knowledge in various forms has been playing a crucial role in improving the learning performance. We study the problem of quantifying the values of domain knowledge in terms of its contribution to the learning performance.
  arXiv  Detail & Related papers  (2020-11-17T06:12:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.