A Generalization Theory for Zero-Shot Prediction

• URL: http://arxiv.org/abs/2507.09128v1
• Date: Sat, 12 Jul 2025 03:37:57 GMT
• Title: A Generalization Theory for Zero-Shot Prediction
• Authors: Ronak Mehta, Zaid Harchaoui,
• Abstract summary: We present a theoretical framework to better understand this approach, called zero-shot prediction.<n>We identify the target quantities that zero-shot prediction aims to learn, or learns in passing, and the key conditional independence relationships that enable its generalization ability.
• Score: 4.1764890353794994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A modern paradigm for generalization in machine learning and AI consists of pre-training a task-agnostic foundation model, generally obtained using self-supervised and multimodal contrastive learning. The resulting representations can be used for prediction on a downstream task for which no labeled data is available. We present a theoretical framework to better understand this approach, called zero-shot prediction. We identify the target quantities that zero-shot prediction aims to learn, or learns in passing, and the key conditional independence relationships that enable its generalization ability.

Related papers

• Global Convergence of Continual Learning on Non-IID Data [51.99584235667152]
  We provide a general and comprehensive theoretical analysis for continual learning of regression models.<n>We establish the almost sure convergence results of continual learning under a general data condition for the first time.
  arXiv  Detail & Related papers  (2025-03-24T10:06:07Z)
• Rethinking Generalizability and Discriminability of Self-Supervised Learning from Evolutionary Game Theory Perspective [43.510860711231544]
  State-of-the-art self-supervised methods tend to enhance either generalizability or discriminability but not both simultaneously.<n>We propose a novel self-supervised learning method that leverages advancements in reinforcement learning to jointly benefit from the general guidance of EGT.
  arXiv  Detail & Related papers  (2024-11-30T17:20:23Z)
• A Practical Theory of Generalization in Selectivity Learning [8.268822578361824]
  query-driven machine learning models have emerged as a promising estimation technique for query selectivities.<n>We bridge the gaps between state-of-the-art (SOTA) theory based on the Probably Approximately Correct (PAC) learning framework.<n>We show that selectivity predictors induced by signed measures are learnable, which relaxes the reliance on probability measures in SOTA theory.
  arXiv  Detail & Related papers  (2024-09-11T05:10:32Z)
• The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction Tuning [86.19804569376333]
  We show that zero-shot generalization happens very early during instruction tuning.<n>We propose a more grounded training data arrangement framework, Test-centric Multi-turn Arrangement.
  arXiv  Detail & Related papers  (2024-06-17T16:40:21Z)
• Learning Latent Graph Structures and their Uncertainty [63.95971478893842]
  We show that minimizing point-prediction losses does not guarantee proper learning of latent relational information.<n>We propose a sampling-based method that solves this joint learning task.
  arXiv  Detail & Related papers  (2024-05-30T10:49:22Z)
• On the Generalization Ability of Unsupervised Pretraining [53.06175754026037]
  Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. This paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase. Our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.
  arXiv  Detail & Related papers  (2024-03-11T16:23:42Z)
• Surprisal Driven $k$-NN for Robust and Interpretable Nonparametric Learning [1.4293924404819704]
  We shed new light on the traditional nearest neighbors algorithm from the perspective of information theory. We propose a robust and interpretable framework for tasks such as classification, regression, density estimation, and anomaly detection using a single model. Our work showcases the architecture's versatility by achieving state-of-the-art results in classification and anomaly detection.
  arXiv  Detail & Related papers  (2023-11-17T00:35:38Z)
• Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks [77.89179552509887]
  We propose a novel prior learning method for advancing generalization and uncertainty estimation in deep neural networks. The key idea is to exploit scalable and structured posteriors of neural networks as informative priors with generalization guarantees. We exhaustively show the effectiveness of this method for uncertainty estimation and generalization.
  arXiv  Detail & Related papers  (2023-07-15T09:24:33Z)
• Masked prediction tasks: a parameter identifiability view [49.533046139235466]
  We focus on the widely used self-supervised learning method of predicting masked tokens. We show that there is a rich landscape of possibilities, out of which some prediction tasks yield identifiability, while others do not.
  arXiv  Detail & Related papers  (2022-02-18T17:09:32Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective [27.93916012334704]
  Unsupervised and self-supervised learning approaches have become a crucial tool to learn representations for downstream prediction tasks. We present a unifying perspective where several such approaches can be viewed as imposing a regularization on the representation via a learnable function using unlabeled data. We propose a discriminative theoretical framework for analyzing the sample complexity of these approaches, which generalizes the framework of (Balcan and Blum, 2010) to allow learnable regularization functions.
  arXiv  Detail & Related papers  (2020-08-06T04:06:04Z)
• Modeling Generalization in Machine Learning: A Methodological and Computational Study [0.8057006406834467]
  We use the concept of the convex hull of the training data in assessing machine learning generalization. We observe unexpectedly weak associations between the generalization ability of machine learning models and all metrics related to dimensionality.
  arXiv  Detail & Related papers  (2020-06-28T19:06:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.