Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

• URL: http://arxiv.org/abs/2607.19608v1
• Date: Tue, 21 Jul 2026 22:24:18 GMT
• Title: Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models
• Abstract summary: We evaluate instruction-tuned Qwen models across sizes.<n>Both standard accuracy and instruction following generally improve with scale.<n>Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings.
• Score: 3.6463026920885824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. We study this across three tasks - multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering - by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears accurate. Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures.

Related papers

• Interactive Task Alignment as a POMDP [17.743513722485876]
  We introduce a general framework for converting specified tasks into underspecified interactions.<n>We validate our user simulator post hoc with a human user study.<n>We show that post-training with supervised fine-tuning and reinforcement learning improves task alignment, but models still lag behind humans in resolving uncertainty through interaction.
  arXiv  Detail & Related papers  (2026-07-17T18:04:38Z)
• Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs [0.17188280334580197]
  Language models are trained to follow instructions, but they are also powerful pattern completers.<n>We construct conversations in which a user instruction to behave in a target way T is opposed by N hardcoded assistant turns demonstrating a competing pattern P.<n>We measure instruction-following (IF) rates in this setting, across 13 models and 16 different instructions, for up to 50 turns.
  arXiv  Detail & Related papers  (2026-05-19T18:32:20Z)
• Silent Commitment Failure in Instruction-Tuned Language Models: Evidence of Governability Divergence Across Architectures [0.0]
  We introduce governability -- the degree to which a model's errors are detectable before output commitment and correctable once detected.<n>We show benchmark accuracy does not predict governability, correction capacity varies independently of detection, and identical governance scaffolds produce opposite effects across models.<n>We propose a Detection and Correction Matrix classifying model-task combinations into four regimes: Governable, Monitor Only, Steer Blind, and Ungovernable.
  arXiv  Detail & Related papers  (2026-03-22T21:50:28Z)
• Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarks [0.6536121591910934]
  We evaluate base and instruction-tuned models on standard math benchmarks, structurally perturbed variants, and domain-shifted tasks.<n>Our results show that base models surpass instruction-tuned variants on the domain-specific MedCalc benchmark.<n>In instruction-tuned models show sharp declines on perturbed datasets, indicating sensitivity to prompt structure over robust reasoning.
  arXiv  Detail & Related papers  (2026-01-19T17:26:49Z)
• Probing for Arithmetic Errors in Language Models [86.8227317662622]
  Internal activations in language models can be used to detect arithmetic errors.<n>We show that simple probes can accurately decode both the model's predicted output and the correct answer from hidden states.<n>We train lightweight error detectors that predict model correctness with over 90% accuracy.
  arXiv  Detail & Related papers  (2025-07-16T16:27:50Z)
• Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance [68.56701216210617]
  In-principle, one would expect models to adapt to the user context better after instruction finetuning.<n>We observe a surprising failure mode: during instruction tuning, the context reliance under knowledge conflicts initially increases as expected, but then gradually decreases.
  arXiv  Detail & Related papers  (2024-10-14T17:57:09Z)
• Instruction Following without Instruction Tuning [87.72635104686275]
  We find two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model.
  arXiv  Detail & Related papers  (2024-09-21T22:36:22Z)
• Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks [101.40633115037983]
  Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions. How to select new tasks to improve the performance and generalizability of IT models remains an open question. We propose active instruction tuning based on prompt uncertainty, a novel framework to identify informative tasks, and then actively tune the models on the selected tasks.
  arXiv  Detail & Related papers  (2023-11-01T04:40:05Z)
• Instruction Position Matters in Sequence Generation with Large Language Models [67.87516654892343]
  Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization. We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
  arXiv  Detail & Related papers  (2023-08-23T12:36:57Z)
• Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering [26.34649731975005]
  Retriever-augmented instruction-following models are attractive alternatives to fine-tuned approaches for question answering (QA) While the model responses tend to be natural and fluent, the additional verbosity makes traditional QA evaluation metrics unreliable for accurately quantifying model performance. We use both automatic and human evaluation to evaluate these models along two dimensions: 1) how well they satisfy the user's information need (correctness) and 2) whether they produce a response based on the provided knowledge (faithfulness)
  arXiv  Detail & Related papers  (2023-07-31T17:41:00Z)
• Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning [74.70157466822612]
  We systematically study the role of task definitions in instruction learning. We find that model performance drops substantially when removing contents describing the task output. We propose two strategies to help models better leverage task instructions.
  arXiv  Detail & Related papers  (2023-06-01T21:11:24Z)
• Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models [80.23791222509644]
  Inconsistent AI models are considered brittle and untrustworthy by human users. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks. We propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets.
  arXiv  Detail & Related papers  (2023-03-28T16:57:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.