In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification

• URL: http://arxiv.org/abs/2605.08295v1
• Date: Fri, 08 May 2026 10:20:39 GMT
• Title: In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification
• Abstract summary: We show that homogeneous labels--even semantically valid ones--collapse accuracy to =12% across six models.<n>The trigger is label-slot content: the model treats tokens occupying the label position as an exhaustive answer vocabulary.
• Score: 4.738949927143789
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While random demonstration labels barely hurt in-context learning (Min et al., 2022), we show that homogeneous labels--even semantically valid ones--collapse accuracy to <=12% across six models (Pythia, Llama, Qwen; 0.8B--8B) and four tasks. The trigger is label-slot content: the model treats tokens occupying the label position as an exhaustive answer vocabulary, with homogeneity as the maximally collapsed case. A novel set-level fixation finding confirms this: when demonstrations carry varied nonsense tokens from {foo,bar,vex,nit,orb}, the model places 42--67% of probability on the demonstrated set while P(dog) remains below 0.2%. This is inconsistent with latent-concept Bayesian accounts (Xie et al., 2022) and reveals that ICL output is constrained vocabulary retrieval--the model binds its output to the demonstrated token inventory regardless of semantic plausibility. The effect generalizes to 4-way classification (0% accuracy across three models, 1B--8B) and multi-token verbalizers ("very positive"), where we decompose fixation into format-level (template adoption) and content-level (polarity override) components that are experimentally dissociable. Mechanistically, per-item paired activation patching on Pythia-1B recovers 98.4% of the gap (95% CI [84%, 112%]), localizing fixation to a layer-7-centered circuit (rank 2/560, 99.8th percentile; 4-fold CV mean 103%). Cross-architecture logit lens on Llama-3.2-1B replicates the encode-then-override trajectory with causal confirmation (top-5 layers: 89% recovery).

Related papers

• Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions [0.0]
  We study whether temperature scaling retains its calibration properties when this assumption is violated.<n>We find that temperature scaling on hard labels consistently underperforms an oracle directly on soft labels.<n>These findings suggest that calibration protocols built on majority-vote labels systematically misstate model reliability wherever label ambiguity is structural.
  arXiv  Detail & Related papers  (2026-07-15T03:52:06Z)
• Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
  We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
  arXiv  Detail & Related papers  (2026-06-21T16:27:24Z)
• Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection [0.0]
  Multi-turn prompt injection follows a known attack path.<n>We show this attack path leaves an activation-level signature in the model's residual stream.
  arXiv  Detail & Related papers  (2026-04-30T17:16:33Z)
• FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction [0.0017904458681854366]
  We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1)<n>Our system achieves 94.3% accuracy with a content effect of 2.85 and a combined score of 41.88 in nested 5-fold cross-validation on the dataset (N=960)<n>Our results suggest that targeted neuro-symbolic integration, applying formal methods precisely where ensemble consensus is lowest, can improve the combined accuracy-plus-content-effect metric used by this task.
  arXiv  Detail & Related papers  (2026-04-20T14:31:00Z)
• Single-Position Intervention Fails: Distributed Output Templates Drive In-Context Learning [0.0]
  Understanding how large language models encode task identity from few-shot demonstrations is a central open problem in mechanistic interpretability.<n>Prior work uses linear probing to localize task representations, reporting high classification accuracy at specific layers.<n>We reveal a striking dissociation: probing accuracy completely fails to predict causal importance.
  arXiv  Detail & Related papers  (2026-04-10T14:49:07Z)
• The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation [1.8345614451086532]
  RLHF-aligned language models exhibit response homogenization on TruthfulQA.<n>40-79% of questions produce a single semantic cluster across 10 i.i.d. samples.
  arXiv  Detail & Related papers  (2026-03-25T09:35:15Z)
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning [49.16227597771663]
  D2Pruner is a framework that combines debiased importance with a structural pruning mechanism.<n>It reduces FLOPs by 74.2% while retaining 99.2% of its original performance.<n>It marks a significant advancement with up to 63. 53% improvement over existing methods.
  arXiv  Detail & Related papers  (2025-12-22T14:42:31Z)
• DiCaP: Distribution-Calibrated Pseudo-labeling for Semi-Supervised Multi-Label Learning [83.94574004953346]
  Semi-supervised multi-label learning aims to leverage unlabeled data to improve the model's performance.<n>Most existing methods assign equal weights to all pseudo-labels regardless of their quality.<n>We propose Distribution-Calibrated Pseudo-labeling (DiCaP), a correctness-aware framework that estimates posterior precision to calibrate pseudo-label weights.
  arXiv  Detail & Related papers  (2025-11-25T11:55:02Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• MarginMatch: Improving Semi-Supervised Learning with Pseudo-Margins [73.17295479535161]
  MarginMatch is a new SSL approach combining consistency regularization and pseudo-labeling. We analyze the behavior of the model on the pseudo-labeled examples as the training progresses to ensure low quality predictions are masked out. We obtain an improvement in error rate over the state-of-the-art of 3.25% on CIFAR-100 with only 25 labels per class and of 3.78% on STL-10 using as few as 4 labels per class.
  arXiv  Detail & Related papers  (2023-08-17T15:19:04Z)
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence [93.91751021370638]
  Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images.
  arXiv  Detail & Related papers  (2020-01-21T18:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.