RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures

• URL: http://arxiv.org/abs/2601.18924v1
• Date: Mon, 26 Jan 2026 19:52:42 GMT
• Title: RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures
• Abstract summary: We introduce RIFT, Reordered Instruction Following Testbed, to assess instruction following by disentangling structure from content.<n>Across 10,000 evaluations spanning six state-of-the-art open-source LLMs, accuracy dropped by up to 72% under jumping conditions.<n>Results reveal structural sensitivity as a fundamental limitation in current architectures.
• Score: 7.812349915277743
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Models (LLMs) are increasingly relied upon for complex workflows, yet their ability to maintain flow of instructions remains underexplored. Existing benchmarks conflate task complexity with structural ordering, making it difficult to isolate the impact of prompt topology on performance. We introduce RIFT, Reordered Instruction Following Testbed, to assess instruction following by disentangling structure from content. Using rephrased Jeopardy! question-answer pairs, we test LLMs across two prompt structures: linear prompts, which progress sequentially, and jumping prompts, which preserve identical content but require non-sequential traversal. Across 10,000 evaluations spanning six state-of-the-art open-source LLMs, accuracy dropped by up to 72% under jumping conditions (compared to baseline), revealing a strong dependence on positional continuity. Error analysis shows that approximately 50% of failures stem from instruction-order violations and semantic drift, indicating that current architectures internalize instruction following as a sequential pattern rather than a reasoning skill. These results reveal structural sensitivity as a fundamental limitation in current architectures, with direct implications for applications requiring non-sequential control flow such as workflow automation and multi-agent systems.

Related papers

• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding [83.0805548642958]
  MTAC-IFBench is a comprehensive benchmark for instruction-following in software development.<n>It features multi-turn progressive software development instructions with diverse constraints spanning 6 primary and 18 secondary categories.<n>MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with their performance degrading rapidly as the interaction session grows longer.
  arXiv  Detail & Related papers  (2026-09-14T03:57:37Z)
• IFHierBench: Hierarchical Instruction Following for Large Language Models [9.572252219147096]
  We introduce IFHierBench, a hierarchical instruction-following benchmark of 600 prompts stratified across four constraint-tree depths and 35 distinct constraints.<n>We find that even the strongest model only marginally exceeds 50% prompt-level accuracy and that accuracy degrades sharply as constraint depth grows.
  arXiv  Detail & Related papers  (2026-07-30T09:26:06Z)
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows [15.552044829103373]
  We present StructureClaw, an artifact-centered workbench for structural engineering requests.<n>We also introduce StructureClaw-Bench, an executable benchmark of 150 controlled scenarios.<n>Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow.
  arXiv  Detail & Related papers  (2026-07-16T12:13:41Z)
• Coupled Hierarchical Search over Topology and Execution for Agentic Workflow Synthesis [64.2820121997882]
  We introduce HierFlow, a training-free, test-time hierarchical search architecture that automates agentic workflow design.<n> HierFlow maximizes efficiency through an intelligent gating module that selectively triggers execution-level searches based on contextual necessity.<n> Comprehensive testing across question answering, mathematical reasoning, and code generation benchmarks confirms that HierFlow consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-05-20T01:56:11Z)
• Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning [13.499744113926505]
  DRAPE is a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT.<n>DRAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines.
  arXiv  Detail & Related papers  (2026-05-11T15:59:06Z)
• The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning [54.67958855362658]
  We propose a dynamic framework that stress tests unlearning using complex structured queries.<n>Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains.<n>Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets.
  arXiv  Detail & Related papers  (2026-03-11T19:51:33Z)
• LSRIF: Logic-Structured Reinforcement Learning for Instruction Following [56.517329105764475]
  We propose a logic-structured training framework LSRIF that explicitly models instruction logic.<n> Experiments show LSRIF brings significant improvements in instruction-following and general reasoning.
  arXiv  Detail & Related papers  (2026-01-10T05:11:38Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• Quantifying Laziness, Decoding Suboptimality, and Context Degradation in Large Language Models [0.4511923587827302]
  Large Language Models (LLMs) often exhibit behavioral artifacts such as laziness, decoded suboptimality, and context degradation.<n>Our results indicate widespread laziness in satisfying complex multi-part instructions.<n>We suggest strategies to reduce laziness and bolster multi-instruction compliance.
  arXiv  Detail & Related papers  (2025-12-19T03:01:59Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models [31.962209251193272]
  Chain-of-thought (CoT) is expected to universally improve capabilities of large language models (LLMs)<n>We propose RAIF, a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling.<n>We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement.
  arXiv  Detail & Related papers  (2025-06-02T08:11:44Z)
• DecIF: Improving Instruction-Following through Meta-Decomposition [9.939860059820917]
  DecIF is a fully autonomous, meta-decomposition guided framework that generates diverse and high-quality instruction-following data.<n>For instruction generation, we guide LLMs to iteratively produce various types of meta-information, which are then combined with response constraints to form semantically rich instructions.<n>For response generation, we decompose each instruction into atomic-level evaluation criteria, enabling rigorous validation and the elimination of inaccurate instruction-response pairs.
  arXiv  Detail & Related papers  (2025-05-20T06:38:28Z)
• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction [68.6543680065379]
  Large language models (LLMs) are vulnerable to prompt injection attacks.<n>We propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-04-29T07:13:53Z)
• AIR: Complex Instruction Generation via Automatic Iterative Refinement [29.639832268719363]
  Current approaches to generating complex instructions are often irrelevant to the current instruction requirements.<n>We propose a novel automatic iterative refinement framework to generate complex instructions with constraints.<n>We construct the AIR-10K dataset with 10K complex instructions and demonstrate that instructions generated with our approach significantly improve the model's ability to follow complex instructions.
  arXiv  Detail & Related papers  (2025-02-25T02:39:57Z)
• RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation [46.237206695937246]
  We propose Retrieval-And-Structuring (RAS), a framework that dynamically constructs query-specific knowledge graphs.<n>On seven knowledge-intensive benchmarks, RAS consistently outperforms strong baselines.<n>Our results demonstrate that dynamic, query-specific knowledge structuring offers a robust path to improving reasoning accuracy and robustness in language model generation.
  arXiv  Detail & Related papers  (2025-02-16T05:01:49Z)
• Constraint Back-translation Improves Complex Instruction Following of Large Language Models [55.60192044049083]
  Large language models (LLMs) struggle to follow instructions with complex constraints in format, length, etc.<n>Previous works conduct post-training on complex instruction-response pairs generated by feeding complex instructions to advanced LLMs.<n>We propose a novel data generation technique, constraint back-translation.
  arXiv  Detail & Related papers  (2024-10-31T17:42:26Z)
• Instruction Position Matters in Sequence Generation with Large Language Models [67.87516654892343]
  Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization. We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
  arXiv  Detail & Related papers  (2023-08-23T12:36:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.