Schema for In-Context Learning

• URL: http://arxiv.org/abs/2510.13905v2
• Date: Thu, 23 Oct 2025 18:04:38 GMT
• Title: Schema for In-Context Learning
• Authors: Pan Chen, Shaohong Chen, Mark Wang, Shi Xuan Leong, Priscilla Fung, Varinia Bernales, Alan Aspuru-Guzik,
• Abstract summary: In-context learning (ICL) enables language models to adapt to new tasks by conditioning on demonstration examples.<n>Inspired by cognitive science, we introduce SCHEMA ACTIVATED IN CONTEXT (SA-ICL)<n>This framework extracts the representation of the building blocks of cognition for the reasoning process instilled from prior examples.<n>We show that SA-ICL consistently boosts performance, up to 36.19 percent, when the single demonstration example is of high quality.
• Score: 0.7850388075652649
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-Context Learning (ICL) enables transformer-based language models to adapt to new tasks by conditioning on demonstration examples. However, traditional example-driven in-context learning lacks explicit modules for knowledge retrieval and transfer at the abstraction level. Inspired by cognitive science, specifically schema theory, which holds that humans interpret new information by activating pre-existing mental frameworks (schemas) to structure understanding, we introduce SCHEMA ACTIVATED IN CONTEXT LEARNING (SA-ICL). This framework extracts the representation of the building blocks of cognition for the reasoning process instilled from prior examples, creating an abstracted schema, a lightweight, structured template of key inferential steps and their relationships, which is then used to augment a model's reasoning process when presented with a novel question. We demonstrate that a broad range of large language models (LLMs) lack the capacity to form and utilize internal schema-based learning representations implicitly, but instead benefit significantly from explicit schema-based scaffolding. Across chemistry and physics questions from the GPQA dataset, our experiments show that SA-ICL consistently boosts performance, up to 36.19 percent, when the single demonstration example is of high quality, which simultaneously reduces reliance on the number of demonstrations and enhances interpretability. SCHEMA ACTIVATED IN CONTEXT LEARNING not only bridges disparate ICL strategies ranging from pattern priming to Chain-of-Thought prompting, but also paves a new path for enhancing human-like reasoning in LLMs.

Related papers

• Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs [100.02824137397464]
  We investigate how Large Language Models adapt their internal representations when encountering inputs of increasing difficulty.<n>We reveal a consistent and quantifiable phenomenon: as task difficulty increases, the last hidden states of LLMs become substantially sparser.<n>This sparsity--difficulty relation is observable across diverse models and domains.
  arXiv  Detail & Related papers  (2026-03-03T18:48:15Z)
• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering [64.62317305868264]
  We present textbfKBQA-R1, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning.<n>Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions.<n>Experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-10T17:45:42Z)
• Fuzzy, Symbolic, and Contextual: Enhancing LLM Instruction via Cognitive Scaffolding [3.553493344868413]
  We study how prompt-level inductive biases influence the cognitive behavior of large language models (LLMs) in instructional dialogue.<n>We introduce a symbolic scaffolding method paired with a short-term memory schema designed to promote adaptive, structured reasoning.<n>Preliminary results show that our full system consistently outperforms baseline variants.
  arXiv  Detail & Related papers  (2025-08-28T20:46:13Z)
• Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning [50.53703102032562]
  Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks.<n>The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood.
  arXiv  Detail & Related papers  (2025-05-16T08:50:42Z)
• Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling [33.66383220833958]
  Large Language Models (LLMs) exhibit In-Context Learning (ICL)<n>ICL enables the model to perform new tasks conditioning only on the examples provided in the context without updating the model's weights.
  arXiv  Detail & Related papers  (2025-01-09T09:45:05Z)
• LLMs Are In-Context Bandit Reinforcement Learners [39.34655336001172]
  Large Language Models (LLMs) excel at in-context learning (ICL), a supervised learning technique that relies on adding annotated examples to the model context.<n>We investigate a contextual bandit version of in-context reinforcement learning (ICRL), where models learn in-context, online, from external reward, instead of supervised data.
  arXiv  Detail & Related papers  (2024-10-07T17:45:00Z)
• Towards More Unified In-context Visual Understanding [74.55332581979292]
  We present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them.
  arXiv  Detail & Related papers  (2023-12-05T06:02:21Z)
• Auto-ICL: In-Context Learning without Human Supervision [93.05202223767463]
  We propose Automatic In-Context Learning framework that enables the model to autonomously generate examples and instructions for problem-solving. With experiments across various models and datasets, results show that model-generated contexts outperform human-annotated contexts.
  arXiv  Detail & Related papers  (2023-11-15T07:37:28Z)
• In-Context Exemplars as Clues to Retrieving from Large Associative Memory [1.2952137350423816]
  In-context learning (ICL) enables large language models (LLMs) to learn patterns from in-context exemplars without training. How to choose exemplars remains unclear due to the lack of understanding of how in-context learning works. Our study sheds new light on the mechanism of ICL by connecting it to memory retrieval.
  arXiv  Detail & Related papers  (2023-11-06T20:13:29Z)
• A Theory of Emergent In-Context Learning as Implicit Structure Induction [8.17811111226145]
  Scaling large language models leads to an emergent capacity to learn in-context from example demonstrations. We argue that in-context learning relies on recombination of compositional operations found in natural language data. We show how in-context learning is supported by a representation of the input's compositional structure.
  arXiv  Detail & Related papers  (2023-03-14T15:24:05Z)
• elBERto: Self-supervised Commonsense Learning for Question Answering [131.51059870970616]
  We propose a Self-supervised Bidirectional Representation Learning of Commonsense framework, which is compatible with off-the-shelf QA model architectures. The framework comprises five self-supervised tasks to force the model to fully exploit the additional training signals from contexts containing rich commonsense. elBERto achieves substantial improvements on out-of-paragraph and no-effect questions where simple lexical similarity comparison does not help.
  arXiv  Detail & Related papers  (2022-03-17T16:23:45Z)
• A Dependency Syntactic Knowledge Augmented Interactive Architecture for End-to-End Aspect-based Sentiment Analysis [73.74885246830611]
  We propose a novel dependency syntactic knowledge augmented interactive architecture with multi-task learning for end-to-end ABSA. This model is capable of fully exploiting the syntactic knowledge (dependency relations and types) by leveraging a well-designed Dependency Relation Embedded Graph Convolutional Network (DreGcn) Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-04T14:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.