The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

• URL: http://arxiv.org/abs/2606.25450v2
• Date: Thu, 25 Jun 2026 16:37:45 GMT
• Title: The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms
• Abstract summary: Traditional evaluations measure a learning algorithm's final performance on an i.i.d. test set, reducing learning to a single aggregate score.<n>This approach obscures a fundamental question: to what extent does learning from a specific example generalize to others?<n>We introduce the Generalization Spectrum, an evaluation framework designed to expose this hidden dimension.
• Score: 48.61395929803806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional evaluations measure a learning algorithm's final performance on an i.i.d. test set, reducing learning to a single aggregate score. This approach obscures a fundamental question: to what extent does learning from a specific example generalize to others? Such per-sample generalization, akin to learning by analogy in human cognition, captures how far the knowledge extracted from one example can transfer, yet remains invisible to standard benchmarks. We introduce the Generalization Spectrum, an evaluation framework designed to expose this hidden dimension. For each training example, we construct a controlled suite of test variants arranged by increasing transfer distance, from exact recall to implementation transfer across languages, context transfer under complete narrative re-framing, category-matched in-domain problems, and an unpaired baseline. By tracking performance across these distances, we reveal not just whether an algorithm learns, but how far that learning extends. We instantiate this framework on competitive programming, using a selection-and-synthesis pipeline seeded with recent problems to mitigate contamination. We first compare three canonical learning paradigms under matched memorization. RL converts memorization into near-transfer more efficiently than SFT-family baselines, while ICL exhibits strong but correspondence-dependent transfer. We then use the Spectrum to diagnose within-family variants. The resulting profiles show that local gains need not expand the generalization radius: abstractions and hints mainly lift local transfer, RFT preserves a stronger far-transfer tail than reference SFT, and self-distillation or hint-assisted RL can reduce far transfer even when local transfer or optimization improves.

Related papers

• Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning [83.67663266193308]
  Class-Incremental Learning (CIL) requires a learning system to learn new classes while retaining previously learned knowledge.<n>In real-world scenarios such as autonomous driving, a system trained on urban roads in sunny weather may later need to operate in rural or highway environments.<n>We propose a unified framework to address domain shift and catastrophic forgetting simultaneously.
  arXiv  Detail & Related papers  (2026-05-09T09:49:13Z)
• SMART: A Spectral Transfer Approach to Multi-Task Learning [9.362937796676855]
  Multi-task learning is effective for related applications, but its performance can deteriorate when the target sample size is small.<n>We propose a singular singular subspace for multi-task linear regression that assumes spectral similarity between the source subspaces and are sparsely aligned the source singular bases.
  arXiv  Detail & Related papers  (2026-04-22T04:00:31Z)
• Learning Admissible Heuristics for A*: Theory and Practice [8.408138419383747]
  deep learning approaches often disregard admissibility and provide limited guarantees on generalization beyond the training data.<n>This paper addresses both of these limitations. First, we pose learning as a constrained optimization problem and introduce Cross-Entropy Admissibility (CEA), a loss function that enforces admissibility during training.<n>On the Rubik's Cube domain, this method yields near-admissibles with significantly stronger than compressed pattern database (PDB) guidances.
  arXiv  Detail & Related papers  (2025-09-26T17:51:26Z)
• Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Test [22.499052329934603]
  We investigate when an LLM memorizes the training data, when its generalization on downstream tasks starts to improve, and what happens if there is a lag between the two.<n>Our study, for the first time, shows that grokking still emerges in pretraining mixture-of-experts (MoE)<n>Our primary discovery is that the pathways evolve from random, non-smooth across layers, instance-specific to more structured and transferable across samples, despite the converged pretraining loss.
  arXiv  Detail & Related papers  (2025-06-26T17:59:58Z)
• Improved Generalization Bounds for Communication Efficient Federated Learning [4.3707341422218215]
  This paper focuses on reducing the communication cost of federated learning by exploring generalization bounds and representation learning. We design a novel Federated Learning with Adaptive Local Steps (FedALS) algorithm based on our generalization bound and representation learning analysis.
  arXiv  Detail & Related papers  (2024-04-17T21:17:48Z)
• A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation [121.0693322732454]
  Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods to enhance CLIP's performance in downstream tasks. We revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP.
  arXiv  Detail & Related papers  (2024-02-06T15:45:27Z)
• FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning [21.088762527081883]
  Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. We explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes.
  arXiv  Detail & Related papers  (2023-09-25T11:54:33Z)
• Mitigating Generation Shifts for Generalized Zero-Shot Learning [52.98182124310114]
  Generalized Zero-Shot Learning (GZSL) is the task of leveraging semantic information (e.g., attributes) to recognize the seen and unseen samples, where unseen classes are not observable during training. We propose a novel Generation Shifts Mitigating Flow framework for learning unseen data synthesis efficiently and effectively. Experimental results demonstrate that GSMFlow achieves state-of-the-art recognition performance in both conventional and generalized zero-shot settings.
  arXiv  Detail & Related papers  (2021-07-07T11:43:59Z)
• Quantifying and Improving Transferability in Domain Generalization [53.16289325326505]
  Out-of-distribution generalization is one of the key challenges when transferring a model from the lab to the real world. We formally define transferability that one can quantify and compute in domain generalization. We propose a new algorithm for learning transferable features and test it over various benchmark datasets.
  arXiv  Detail & Related papers  (2021-06-07T14:04:32Z)
• Graph Sampling Based Deep Metric Learning for Generalizable Person Re-Identification [114.56752624945142]
  We argue that the most popular random sampling method, the well-known PK sampler, is not informative and efficient for deep metric learning. We propose an efficient mini batch sampling method called Graph Sampling (GS) for large-scale metric learning.
  arXiv  Detail & Related papers  (2021-04-04T06:44:15Z)
• Fast Few-Shot Classification by Few-Iteration Meta-Learning [173.32497326674775]
  We introduce a fast optimization-based meta-learning method for few-shot classification. Our strategy enables important aspects of the base learner objective to be learned during meta-training. We perform a comprehensive experimental analysis, demonstrating the speed and effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-10-01T15:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.