Statistical learning does not always entail knowledge

• URL: http://arxiv.org/abs/2501.01963v1
• Date: Tue, 17 Dec 2024 19:51:32 GMT
• Title: Statistical learning does not always entail knowledge
• Authors: Daniel Andrés Díaz-Pachón, H. Renata Gallegos, Ola Hössjer, J. Sunil Rao,
• Abstract summary: We study learning and knowledge acquisition of an agent about a proposition that is either true or false. It is assumed that data provide details about a number of features that are relevant to the proposition. We show that full learning is sometimes not possible and full knowledge acquisition is never possible when the number of extracted features is too small.
• Score: 1.8749305679160366
• License:
• Abstract: In this paper, we study learning and knowledge acquisition (LKA) of an agent about a proposition that is either true or false. We use a Bayesian approach, where the agent receives data to update his beliefs about the proposition according to a posterior distribution. The LKA is formulated in terms of active information, with data representing external or exogenous information that modifies the agent's beliefs. It is assumed that data provide details about a number of features that are relevant to the proposition. We show that this leads to a Gibbs distribution posterior, which is in maximum entropy relative to the prior, conditioned on the side constraints that the data provide in terms of the features. We demonstrate that full learning is sometimes not possible and full knowledge acquisition is never possible when the number of extracted features is too small. We also distinguish between primary learning (receiving data about features of relevance for the proposition) and secondary learning (receiving data about the learning of another agent). We argue that this type of secondary learning does not represent true knowledge acquisition. Our results have implications for statistical learning algorithms, and we claim that such algorithms do not always generate true knowledge. The theory is illustrated with several examples.

Related papers

• Collaborative Learning with Different Labeling Functions [7.228285747845779]
  We study a variant of Collaborative PAC Learning, in which we aim to learn an accurate classifier for each of the $n$ data distributions. We show that, when the data distributions satisfy a weaker realizability assumption, sample-efficient learning is still feasible.
  arXiv  Detail & Related papers  (2024-02-16T04:32:22Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction [51.68385617116854]
  Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data.
  arXiv  Detail & Related papers  (2023-09-25T17:37:20Z)
• Reinforcement Learning from Passive Data via Latent Intentions [86.4969514480008]
  We show that passive data can still be used to learn features that accelerate downstream RL. Our approach learns from passive data by modeling intentions. Our experiments demonstrate the ability to learn from many forms of passive data, including cross-embodiment video data and YouTube videos.
  arXiv  Detail & Related papers  (2023-04-10T17:59:05Z)
• A Survey of Learning on Small Data: Generalization, Optimization, and Challenge [101.27154181792567]
  Learning on small data that approximates the generalization ability of big data is one of the ultimate purposes of AI. This survey follows the active sampling theory under a PAC framework to analyze the generalization error and label complexity of learning on small data. Multiple data applications that may benefit from efficient small data representation are surveyed.
  arXiv  Detail & Related papers  (2022-07-29T02:34:19Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)
• A Bayesian Framework for Information-Theoretic Probing [51.98576673620385]
  We argue that probing should be seen as approximating a mutual information. This led to the rather unintuitive conclusion that representations encode exactly the same information about a target task as the original sentences. This paper proposes a new framework to measure what we term Bayesian mutual information.
  arXiv  Detail & Related papers  (2021-09-08T18:08:36Z)
• A Bayesian Approach to (Online) Transfer Learning: Theory and Algorithms [6.193838300896449]
  We study transfer learning from a Bayesian perspective, where a parametric statistical model is used. Specifically, we study three variants of transfer learning problems, instantaneous, online, and time-variant transfer learning. For each problem, we define an appropriate objective function, and provide either exact expressions or upper bounds on the learning performance. Examples show that the derived bounds are accurate even for small sample sizes.
  arXiv  Detail & Related papers  (2021-09-03T08:43:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.