Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

• URL: http://arxiv.org/abs/2608.22753v1
• Date: Mon, 24 Aug 2026 03:22:45 GMT
• Title: Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models
• Abstract summary: Large language models (LLMs) excel at text understanding and generation, yet struggle to reliably understand and apply externally provided procedural rules at scale.<n>We introduce RuleWorld, a large-scale benchmark that reformulates rules as globally reusable abstract units rather than instance-specific facts.<n>We propose DynaRule, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process.
• Score: 33.61747923589224
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) excel at text understanding and generation, yet still struggle to reliably understand and apply externally provided procedural rules at scale. To evaluate this capability, we introduce RuleWorld, a large-scale benchmark that reformulates rules as globally reusable abstract units rather than instance-specific facts. In RuleWorld, several scenarios, including single-rule, parallel multi-rule, and multi-hop reasoning, are settled for comprehensive evaluation. We further propose DynaRule, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process. Specifically, DynaRule employs Stacked Step-Level Attention Training with a special <search> token to enable dynamic rule re-attention and updating during inference. In this way, the model can re-attend to the most relevant rules at each step, dynamically replacing outdated ones to support more stable multi-step reasoning. Experiments on RuleWorld show that existing LLMs face challenges under large rule pools, while DynaRule improves average QA accuracy by up to 19 points and achieves over 85% Recall@1 at 10K rules, outperforming strong baselines by large margins. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/Beyond-Factual-Knowledge.

Related papers

• RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models [39.01774351778858]
  This paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning.<n>We show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score.
  arXiv  Detail & Related papers  (2026-08-27T09:05:11Z)
• Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models [56.355176278601476]
  Large language models store not only isolated facts but also rules that support reasoning across symbolic expressions, natural language explanations, and concrete instances.<n>Most model editing methods are built for fact-level knowledge, assuming that a target edit can be achieved through a localized intervention.<n>We investigate this problem through a mechanistic study of rule-level knowledge editing.
  arXiv  Detail & Related papers  (2026-04-09T14:22:19Z)
• Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs [23.958458849973248]
  We study length generalization in multi-task settings and propose Meta Rule-Following Fine-Tuning (Meta-RFFT) as the first framework enabling robust cross-task length generalization.<n>After training on a large number of tasks and instances, our models achieve remarkable length generalization ability on unseen tasks with minimal fine-tuning or one-shot prompting.
  arXiv  Detail & Related papers  (2025-02-17T07:54:50Z)
• RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios [58.90106984375913]
  RuleArena is a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning.<n> Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions.
  arXiv  Detail & Related papers  (2024-12-12T06:08:46Z)
• RNR: Teaching Large Language Models to Follow Roles and Rules [153.6596303205894]
  We propose model, an automated data generation pipeline that generates diverse roles and rules from existing IFT instructions. This data can then be used to train models that follow complex system prompts. Our framework significantly improves role and rule following capability in large language models.
  arXiv  Detail & Related papers  (2024-09-10T06:07:32Z)
• Symbolic Working Memory Enhances Language Models for Complex Rule Application [87.34281749422756]
  Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning. We propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. Our framework iteratively performs symbolic rule grounding and LLM-based rule implementation.
  arXiv  Detail & Related papers  (2024-08-24T19:11:54Z)
• Can LLMs Follow Simple Rules? [28.73820874333199]
  Rule-following Language Evaluation Scenarios (RuLES) is a framework for measuring rule-following ability in Large Language Models. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. We show that almost all current models struggle to follow scenario rules, even on straightforward test cases.
  arXiv  Detail & Related papers  (2023-11-06T08:50:29Z)
• ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning [107.61997887260056]
  We propose a novel framework, ChatRule, unleashing the power of large language models for mining logical rules over knowledge graphs. Specifically, the framework is initiated with an LLM-based rule generator, leveraging both the semantic and structural information of KGs. To refine the generated rules, a rule ranking module estimates the rule quality by incorporating facts from existing KGs.
  arXiv  Detail & Related papers  (2023-09-04T11:38:02Z)
• Building Rule Hierarchies for Efficient Logical Rule Learning from Knowledge Graphs [20.251630903853016]
  We propose new methods for pruning unpromising rules using rule hierarchies. We show that the application of HPMs is effective in removing unpromising rules.
  arXiv  Detail & Related papers  (2020-06-29T16:33:30Z)
• Towards Learning Instantiated Logical Rules from Knowledge Graphs [20.251630903853016]
  We present GPFL, a probabilistic learner rule optimized to mine instantiated first-order logic rules from knowledge graphs. GPFL utilizes a novel two-stage rule generation mechanism that first generalizes extracted paths into templates that are acyclic abstract rules. We reveal the presence of overfitting rules, their impact on the predictive performance, and the effectiveness of a simple validation method filtering out overfitting rules.
  arXiv  Detail & Related papers  (2020-03-13T00:32:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.