Emergence of Context Characteristics Sensitivity in Large Language Models

• URL: http://arxiv.org/abs/2606.09525v1
• Date: Mon, 08 Jun 2026 14:11:07 GMT
• Title: Emergence of Context Characteristics Sensitivity in Large Language Models
• Abstract summary: We study how models' sensitivity to context characteristics shifts across successive instruction fine-tuning stages.<n>Experiments show that SFT makes models more likely to use contexts that are easy to understand.<n>Our findings reveal that context usage is actively reshaped at each IFT stage.
• Score: 34.75303234486382
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: During instruction fine-tuning (IFT), large language models (LLMs) learn to follow instructions by using the provided context to answer a query. While prior work has studied how context characteristics correlate with context usage by the LLM, this analysis has been limited to inference time, leaving open how these relationships are acquired in the first place. Here, we measure how models' sensitivity to such characteristics shifts across successive IFT stages: supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with verifiable rewards (RLVR). Experiments across four models and three datasets show that SFT makes models more likely to use contexts that are easy to understand, such as containing high length, context-query similarity, and fluency. Post-SFT dynamics may either reinforce or resolve these preferences depending on the training dataset. Our findings reveal that context usage is actively reshaped at each IFT stage, and designing a balanced IFT dataset is important in ensuring robust context utilization of instruction-tuned models.

Related papers

• VIP-COP: Context Optimization for Tabular Foundation Models [23.854424850102273]
  VIP-COP estimates the Value of Importance for Prediction of training examples and features for hard Context OPtimization for TFMs.<n>Its explicit selection mechanism suppresses noise and isolates influential data, enabling the model to also benefit from data augmentation.<n> VIP-COP is (i) fast, boosting performance often within minutes of optimization; (ii) budget-aware and any-time, improving with additional test-time compute; (iii) model-aware yet fully black-box, requiring no access to model internals; (iv) interpretable, identifying discrete Very Important Predictors'' (sample
  arXiv  Detail & Related papers  (2026-05-13T02:28:31Z)
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models [102.20309135516186]
  Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
  arXiv  Detail & Related papers  (2026-03-12T17:57:50Z)
• SIEVE: Sample-Efficient Parametric Learning from Natural Language [66.46967795931559]
  We propose SIEVE, a method for sample-efficient parametric learning from natural language context.<n>We use a novel synthetic data generation pipeline, SIEVE-GEN, that leverages the insight that context is decomposable.<n>Our results show that SIEVE outperforms prior context distillation methods using just three query examples.
  arXiv  Detail & Related papers  (2026-02-02T18:52:52Z)
• Context-level Language Modeling by Learning Predictive Context Embeddings [79.00607069677393]
  We introduce textbfContextLM, a framework that augments standard pretraining with an inherent textbfnext-context prediction objective.<n>This mechanism trains the model to learn predictive representations of multi-token contexts, leveraging error signals derived from future token chunks.<n>Experiments on the GPT2 and Pythia model families, scaled up to $1.5$B parameters, show that ContextLM delivers consistent improvements in both perplexity and downstream task performance.
  arXiv  Detail & Related papers  (2025-10-23T07:09:45Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models [12.500777267361102]
  We introduce a novel textbfpreference-textbforiented supervised textbffine-textbftuning approach, namely PoFT.<n>The intuition is to boost SFT by imposing a particular preference: textitfavoring the target model over aligned LLMs on the same SFT data.<n>PoFT achieves stable and consistent improvements over the SFT baselines across different training datasets and base models.
  arXiv  Detail & Related papers  (2024-12-17T12:49:14Z)
• On the Loss of Context-awareness in General Instruction Fine-tuning [101.03941308894191]
  We investigate the loss of context awareness after supervised fine-tuning.<n>We find that the performance decline is associated with a bias toward different roles learned during conversational instruction fine-tuning.<n>We propose a metric to identify context-dependent examples from general instruction fine-tuning datasets.
  arXiv  Detail & Related papers  (2024-11-05T00:16:01Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context.<n>We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters.<n>We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe [36.74756622715754]
  Large language models (LLMs) undergo instruction tuning to acquire instruction-following capabilities.<n>Efforts to improve instruction tuning often focus on higher-quality supervised fine-tuning datasets.<n>We propose SFTMix, a novel Mixup-based recipe that elevates LLM instruction tuning without relying on well-curated datasets.
  arXiv  Detail & Related papers  (2024-10-07T17:52:21Z)
• Instruction Tuning for Large Language Models: A Survey [52.86322823501338]
  We make a systematic review of the literature, including the general methodology of supervised fine-tuning (SFT)<n>We also review the potential pitfalls of SFT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies.
  arXiv  Detail & Related papers  (2023-08-21T15:35:16Z)
• Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning [23.971206470486468]
  We present a framework that converts relevant historical facts into prompts and generates ranked predictions using token probabilities. Surprisingly, we observe that LLMs, out-of-the-box, perform on par with state-of-the-art TKG models. We also discover that using numerical indices instead of entity/relation names, does not significantly affect the performance.
  arXiv  Detail & Related papers  (2023-05-17T23:50:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.