RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

• URL: http://arxiv.org/abs/2608.26832v1
• Date: Thu, 27 Aug 2026 09:05:11 GMT
• Title: RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
• Abstract summary: This paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning.<n>We show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score.
• Score: 39.01774351778858
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.

Related papers

• Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models [33.61747923589224]
  Large language models (LLMs) excel at text understanding and generation, yet struggle to reliably understand and apply externally provided procedural rules at scale.<n>We introduce RuleWorld, a large-scale benchmark that reformulates rules as globally reusable abstract units rather than instance-specific facts.<n>We propose DynaRule, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process.
  arXiv  Detail & Related papers  (2026-08-24T03:22:45Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• Logic-Regularized Verifier Elicits Reasoning from LLMs [63.65875399266337]
  We propose LOVER, an unsupervised verifier regularized by logical rules.<n>Lover treats theverifier as a binary latent variable, utilizinginternal activations and enforcing three logical constraints.<n>Lover significantly outperforms unsupervised baselines.
  arXiv  Detail & Related papers  (2026-05-07T09:03:49Z)
• SLogic: Subgraph-Informed Logical Rule Learning for Knowledge Graph Completion [2.8845104295670017]
  We introduce SLogic, a framework that assigns query-dependent scores to logical rules.<n>The core of SLogic is a scoring function that utilizes the subgraph centered on a query's head entity.<n>By leveraging local subgraph context, SLogic consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-09-30T20:59:22Z)
• Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs [23.958458849973248]
  We study length generalization in multi-task settings and propose Meta Rule-Following Fine-Tuning (Meta-RFFT) as the first framework enabling robust cross-task length generalization.<n>After training on a large number of tasks and instances, our models achieve remarkable length generalization ability on unseen tasks with minimal fine-tuning or one-shot prompting.
  arXiv  Detail & Related papers  (2025-02-17T07:54:50Z)
• LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models [87.49676980090555]
  Large Language Models (LLMs) have demonstrated notable capabilities across various tasks, showcasing complex problem-solving abilities. We introduce LogicGame, a novel benchmark designed to evaluate the comprehensive rule understanding, execution, and planning capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-08-28T13:16:41Z)
• Symbolic Working Memory Enhances Language Models for Complex Rule Application [87.34281749422756]
  Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning. We propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. Our framework iteratively performs symbolic rule grounding and LLM-based rule implementation.
  arXiv  Detail & Related papers  (2024-08-24T19:11:54Z)
• Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs [87.34281749422756]
  Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. We propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic.
  arXiv  Detail & Related papers  (2024-02-18T03:38:51Z)
• Can LLMs Follow Simple Rules? [28.73820874333199]
  Rule-following Language Evaluation Scenarios (RuLES) is a framework for measuring rule-following ability in Large Language Models. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. We show that almost all current models struggle to follow scenario rules, even on straightforward test cases.
  arXiv  Detail & Related papers  (2023-11-06T08:50:29Z)
• Building Rule Hierarchies for Efficient Logical Rule Learning from Knowledge Graphs [20.251630903853016]
  We propose new methods for pruning unpromising rules using rule hierarchies. We show that the application of HPMs is effective in removing unpromising rules.
  arXiv  Detail & Related papers  (2020-06-29T16:33:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.