Exemplar Retrieval Without Overhypothesis Induction: Limits of Distributional Sequence Learning in Early Word Learning

• URL: http://arxiv.org/abs/2604.05243v1
• Date: Mon, 06 Apr 2026 23:11:25 GMT
• Title: Exemplar Retrieval Without Overhypothesis Induction: Limits of Distributional Sequence Learning in Early Word Learning
• Abstract summary: Children learn that shape is the kind of feature that tends to define object categories.<n>We trained autoregressive transformer language models on synthetic corpora.<n>Models rely on frame-to-feature template matching rather than structured noun-to-domain-to-feature abstraction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Children do not simply learn that balls are round and blocks are square. They learn that shape is the kind of feature that tends to define object categories -- a second-order generalisation known as an overhypothesis [1, 2]. What kind of learning mechanism is sufficient for this inductive leap? Methods: We trained autoregressive transformer language models (3.4M-25.6M parameters) on synthetic corpora in which shape is the stable feature dimension across categories, with eight conditions controlling for alternative explanations. Results: Across 120 pre-registered runs evaluated on a 1,040-item wug test battery, every model achieved perfect first-order exemplar retrieval (100%) while second-order generalisation to novel nouns remained at chance (50-52%), a result confirmed by equivalence testing. A feature-swap diagnostic revealed that models rely on frame-to-feature template matching rather than structured noun-to-domain-to-feature abstraction. Conclusions: These results reveal a clear limitation of autoregressive distributional sequence learning under developmental-scale training conditions.

Related papers

• Causal dictionary learning reveals and validates transcription-factor binding features in genomic language models [5.237172334460829]
  We introduce a framework that combines sparse dictionary learning with causal intervention to extract, validate, and causally test interpretable features in genomic foundation models.<n>We show that the naive validation of such features against position weight matrices is severely confounded by GC composition and repetitive elements.<n>We establish that specific features are emphcausally used to represent cell-type-specific TF binding, not merely motif presence.
  arXiv  Detail & Related papers  (2026-07-21T22:54:46Z)
• AI Mental Models: Learned Intuition and Deliberation in a Bounded Neural Architecture [0.0]
  This paper asks whether a bounded neural architecture can exhibit a meaningful division of labor between intuition and deliberation on a syllogistic reasoning benchmark.<n>Experiment 1 evaluates a direct neural baseline for predicting full 9-way human response distributions under 5-fold cross-validation.<n>Experiment 2 introduces a bounded dual-path architecture with separate intuition and deliberation pathways, motivated by computational mental-model theory.
  arXiv  Detail & Related papers  (2026-03-23T20:44:38Z)
• Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling [60.63703438729223]
  We show how different architectures and training methods affect model multi-step reasoning capabilities.<n>We confirm that increasing model depth plays a crucial role for sequential computations.
  arXiv  Detail & Related papers  (2025-08-22T18:57:08Z)
• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors [61.92704516732144]
  We show that the most robust features for correctness prediction are those that play a distinctive causal role in the model's behavior.<n>We propose two methods that leverage causal mechanisms to predict the correctness of model outputs.
  arXiv  Detail & Related papers  (2025-05-17T00:31:39Z)
• Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models [5.015875080287129]
  Self-supervised word embedding algorithms such as word2vec provide a minimal setting for studying representation learning in language modeling.<n>We show that both the training dynamics and the final performance on downstream tasks are empirically very similar to those of word2vec.
  arXiv  Detail & Related papers  (2025-02-14T02:16:48Z)
• Geometric Prior Guided Feature Representation Learning for Long-Tailed Classification [47.09355487357069]
  We propose to leverage the geometric information of the feature distribution of the well-represented head class to guide the model to learn the underlying distribution of the tail class. It aims to make the perturbed features cover the underlying distribution of the tail class as much as possible, thus improving the model's generalization performance in the test domain.
  arXiv  Detail & Related papers  (2024-01-21T09:16:29Z)
• Unleashing the power of Neural Collapse for Transferability Estimation [42.09673383041276]
  Well-trained models exhibit the phenomenon of Neural Collapse. We propose a novel method termed Fair Collapse (FaCe) for transferability estimation. FaCe yields state-of-the-art performance on different tasks including image classification, semantic segmentation, and text classification.
  arXiv  Detail & Related papers  (2023-10-09T14:30:10Z)
• Training Trajectories of Language Models Across Scales [99.38721327771208]
  Scaling up language models has led to unprecedented performance gains. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors?
  arXiv  Detail & Related papers  (2022-12-19T19:16:29Z)
• Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality [84.94877848357896]
  Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples. We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
  arXiv  Detail & Related papers  (2022-11-28T17:36:41Z)
• Semantic Self-adaptation: Enhancing Generalization with a Single Sample [45.111358665370524]
  We propose a self-adaptive approach for semantic segmentation. It fine-tunes the parameters of convolutional layers to the input image using consistency regularization. Our empirical study suggests that self-adaptation may complement the established practice of model regularization at training time.
  arXiv  Detail & Related papers  (2022-08-10T12:29:01Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Linguistically inspired morphological inflection with a sequence to sequence model [19.892441884896893]
  Our research question is whether a neural network would be capable of learning inflectional morphemes for inflection production. We are using an inflectional corpus and a single layer seq2seq model to test this hypothesis. Our character-morpheme-based model creates inflection by predicting the stem character-to-character and the inflectional affixes as character blocks.
  arXiv  Detail & Related papers  (2020-09-04T08:58:42Z)
• Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? [41.649440404203595]
  We introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits.
  arXiv  Detail & Related papers  (2020-04-30T14:48:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.