Distinct Computations Emerge From Compositional Curricula in In-Context Learning

• URL: http://arxiv.org/abs/2506.13253v1
• Date: Mon, 16 Jun 2025 08:49:42 GMT
• Title: Distinct Computations Emerge From Compositional Curricula in In-Context Learning
• Authors: Jin Hwa Lee, Andrew K. Lampinen, Aaditya K. Singh, Andrew M. Saxe,
• Abstract summary: We investigate how presenting a compositional subtask curriculum in context may alter the computations a transformer learns.<n>We show that models trained with a subtask curriculum can perform zero-shot inference on unseen compositional tasks.<n>We find that the models employ diverse strategies modulated by the specific curriculum design.
• Score: 14.14936155458072
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In-context learning (ICL) research often considers learning a function in-context through a uniform sample of input-output pairs. Here, we investigate how presenting a compositional subtask curriculum in context may alter the computations a transformer learns. We design a compositional algorithmic task based on the modular exponential-a double exponential task composed of two single exponential subtasks and train transformer models to learn the task in-context. We compare (a) models trained using an in-context curriculum consisting of single exponential subtasks and, (b) models trained directly on the double exponential task without such a curriculum. We show that models trained with a subtask curriculum can perform zero-shot inference on unseen compositional tasks and are more robust given the same context length. We study how the task and subtasks are represented across the two training regimes. We find that the models employ diverse strategies modulated by the specific curriculum design.

Related papers

• In-context Learning in Presence of Spurious Correlations [8.055478206164105]
  We study the possibility of training an in-context learner for classification tasks involving spurious features. We find that the conventional approach of training in-context learners is susceptible to spurious features. We propose a novel technique to train such a learner for a given classification task.
  arXiv  Detail & Related papers  (2024-10-04T04:26:36Z)
• Dual Operating Modes of In-Context Learning [8.664657381613695]
  In-context learning (ICL) exhibits dual operating modes: task learning, and task retrieval. Recent theoretical work investigates various mathematical models to analyze ICL. We introduce a probabilistic model, with which one can explain the dual operating modes of ICL simultaneously.
  arXiv  Detail & Related papers  (2024-02-29T03:06:10Z)
• Limits of Transformer Language Models on Learning to Compose Algorithms [77.2443883991608]
  We evaluate training LLaMA models and prompting GPT-4 and Gemini on four tasks demanding to learn a composition of several discrete sub-tasks. Our results indicate that compositional learning in state-of-the-art Transformer language models is highly sample inefficient.
  arXiv  Detail & Related papers  (2024-02-08T16:23:29Z)
• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes [67.06980111346245]
  In-context learning refers to the ability of a model to condition on a prompt sequence consisting of in-context examples. We show that standard Transformers can be trained from scratch to perform in-context learning of linear functions. We also show that we can train Transformers to in-context learn more complex function classes with performance that matches or exceeds task-specific learning algorithms.
  arXiv  Detail & Related papers  (2022-08-01T18:01:40Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Combining Modular Skills in Multitask Learning [149.8001096811708]
  A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to new tasks. In this work, we assume each task is associated with a subset of latent discrete skills from a (potentially small) inventory. We find that the modular design of a network significantly increases sample efficiency in reinforcement learning and few-shot generalisation in supervised learning.
  arXiv  Detail & Related papers  (2022-02-28T16:07:19Z)
• Active Multi-Task Representation Learning [50.13453053304159]
  We give the first formal study on resource task sampling by leveraging the techniques from active learning. We propose an algorithm that iteratively estimates the relevance of each source task to the target task and samples from each source task based on the estimated relevance.
  arXiv  Detail & Related papers  (2022-02-02T08:23:24Z)
• Learning to Follow Language Instructions with Compositional Policies [22.778677208048475]
  We propose a framework that learns to execute natural language instructions in an environment consisting of goal-reaching tasks. We train a reinforcement learning agent to learn value functions that can be subsequently composed through a Boolean algebra. We fine-tune a seq2seq model pretrained on web-scale corpora to map language to logical expressions.
  arXiv  Detail & Related papers  (2021-10-09T21:28:26Z)
• On the relationship between disentanglement and multi-task learning [62.997667081978825]
  We take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter sharing. We show that disentanglement appears naturally during the process of multi-task neural network training.
  arXiv  Detail & Related papers  (2021-10-07T14:35:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.