ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints

• URL: http://arxiv.org/abs/2406.04046v3
• Date: Sun, 02 Mar 2025 23:24:43 GMT
• Title: ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints
• Abstract summary: Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational AI problems.<n>We introduce a new diagnostic benchmark, ActionReasoningBench, which encompasses 8 domains and includes questions for up to 19 action sequences.<n>This benchmark rigorously evaluates Large Language Models (LLMs) across six key RAC dimensions.
• Score: 31.90180597239974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational AI problems, such as the frame problem. It has driven advancements in AI fields, such as non-monotonic and commonsense reasoning. RAC remains crucial for AI systems that operate in dynamic environments, engage in interactive scenarios, or rely on commonsense reasoning. Despite substantial advances made by Large Language Models (LLMs) in various AI domains, their performance in RAC remains underexplored. To address this gap, we introduce a new diagnostic benchmark, ActionReasoningBench, which encompasses 8 domains and includes questions for up to 19 action sequences. This benchmark rigorously evaluates LLMs across six key RAC dimensions: Fluent Tracking, State Tracking, Action Executability, Effects of Actions, Numerical RAC, and Composite Questions. LLMs demonstrate average accuracy rates of 73.55%, 65.63%, 58.73%, and 62.38% on the former four dimensions, which are frequently discussed in RAC literature. However, the performance on the latter two dimensions, which introduce complex and novel reasoning questions, the average performance of LLMs is lowered to 33.16% and 51.19%, respectively, reflecting a 17.9% performance decline. We also introduce new ramification constraints to capture the indirect effects of actions, providing deeper insights into RAC challenges. Our evaluation of state-of-the-art LLMs, including both open-source and commercial models, reveals challenges across all RAC dimensions, particularly in handling ramifications, with GPT-4o failing to solve any question and o1-preview achieving a score of only 18.4%.

Related papers

• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild [18.82003667637388]
  Evaluation harnesses orchestrate model evaluation by managing model invocation, data loading, metric computation, and result reporting.<n>We present an empirical study of 57 evaluation harnesses, deriving a five-stage harness model and classifying 16,560 issues by workflow stage and root cause.
  arXiv  Detail & Related papers  (2026-05-22T20:54:30Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• ProRAC: A Neuro-symbolic Method for Reasoning about Actions with LLM-based Progression [3.7838089347652244]
  We propose ProRAC, a neuro-symbolic framework to tackle RAC problems.<n>ProRAC extracts RAC elements including actions and questions from the problem.<n>We evaluate ProRAC on several RAC benchmarks, and the results demonstrate that our approach achieves strong performance.
  arXiv  Detail & Related papers  (2025-11-19T03:20:06Z)
• Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs [49.584982680596546]
  We find that emergent misalignment (EM) emerges in in-context learning (ICL)<n>We find that it does: across three datasets, three frontier models produce broadly misaligned responses at rates between 2% and 17% given 64 narrow in-context examples, and up to 58% with 256 examples.
  arXiv  Detail & Related papers  (2025-10-13T11:23:56Z)
• BEAR: Benchmarking and Enhancing Multimodal Language Models for Atomic Embodied Capabilities [61.173773299032746]
  Embodied capabilities refer to a suite of fundamental abilities for an agent to perceive, comprehend, and interact with the physical world.<n>We introduce BEAR, a comprehensive and fine-grained benchmark that evaluates MLLMs on atomic embodied capabilities.<n> BEAR comprises 4,469 interleaved image-video-text entries across 14 domains in 6 categories, including tasks from low-level pointing, trajectory understanding, spatial reasoning, to high-level planning.<n>We propose BEAR-Agent, a multimodal conversable agent that integrates pretrained vision models to strengthen MLLM perception, 3D understanding, and planning capabilities.
  arXiv  Detail & Related papers  (2025-10-09T19:18:36Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogs [8.581146564012856]
  This study proposes a comprehensive evaluation framework to quantify the behavior gap between AI agents and human experts.<n>Our findings reveal that this behavior gap is a critical factor negatively impacting the performance of LLM agents.<n>For the most complex task in our study, even the GPT-4o-based agent exhibits low alignment with human behavior.
  arXiv  Detail & Related papers  (2025-06-13T22:36:41Z)
• Affordance Benchmark for MLLMs [38.62884479364572]
  We introduce **A4Bench**, a novel benchmark designed to evaluate the affordance perception abilities of MLLMs across two dimensions.<n>We evaluate 17 MLLMs (nine proprietary and eight open-source) and compare them to human performance.<n>Results show that proprietary models generally outperform open-source ones, yet all models perform far below humans.
  arXiv  Detail & Related papers  (2025-06-01T08:26:34Z)
• Evaluating the Sensitivity of LLMs to Prior Context [2.377922603550519]
  Large language models (LLMs) are increasingly deployed in multi-turn dialogue and other sustained interactive scenarios.<n>We introduce a novel set of benchmarks that vary the volume and nature of prior context to measure sensitivity to contextual variations.<n>Our findings reveal that LLM performance on multiple-choice questions can degrade dramatically in multi-turn interactions.
  arXiv  Detail & Related papers  (2025-05-29T16:09:32Z)
• IRCopilot: Automated Incident Response with Large Language Models [18.2902546324428]
  Large Language Models (LLMs) have shown great potential in early threat detection.<n>Their capabilities remain limited when it comes to automated incident response after an intrusion.<n>We propose IRCopilot, a novel framework for automated incident response powered by LLMs.
  arXiv  Detail & Related papers  (2025-05-27T09:29:11Z)
• Assessing Judging Bias in Large Reasoning Models: An Empirical Study [99.86300466350013]
  Large Reasoning Models (LRMs) like DeepSeek-R1 and OpenAI-o1 have demonstrated remarkable reasoning capabilities. We present a benchmark comparing judging biases between LLMs and LRMs across both subjective preference-alignment datasets and objective fact-based datasets.
  arXiv  Detail & Related papers  (2025-04-14T07:14:27Z)
• Question Tokens Deserve More Attention: Enhancing Large Language Models without Training through Step-by-Step Reading and Question Attention Recalibration [0.36561146074362716]
  Large Language Models (LLMs) often struggle with tasks that require a deep understanding of complex questions. This work investigates the limitations of current LLMs in question comprehension. We propose a family of prompt-based strategies that guide LLMs to incrementally process question tokens and align their reasoning with the input structure.
  arXiv  Detail & Related papers  (2025-04-13T02:10:18Z)
• Can AI Master Construction Management (CM)? Benchmarking State-of-the-Art Large Language Models on CM Certification Exams [2.897171041611256]
  This study introduces CMExamSet, a benchmarking dataset comprising 689 authentic multiple-choice questions from four nationally accredited CM certification exams. Results indicate that GPT-4o and Claude 3.7 surpass typical human pass thresholds (70%), with average accuracies of 82% and 83%, respectively. conceptual misunderstandings are the most common, underscoring the need for enhanced domain-specific reasoning models.
  arXiv  Detail & Related papers  (2025-04-04T18:13:45Z)
• Effectively Controlling Reasoning Models through Thinking Intervention [38.77100471547442]
  Reasoning-enhanced large language models explicitly generate intermediate reasoning steps prior to generating final answers. This emerging generation framework offers a unique opportunity for more fine-grained control over model behavior. We propose Thinking Intervention, a novel paradigm designed to explicitly guide the internal reasoning processes of LLMs.
  arXiv  Detail & Related papers  (2025-03-31T17:50:13Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step. Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy. We propose Reasoning-Driven Process Reward Modeling (R-PRM) R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models [9.090588805667263]
  We propose SECURA: Sigmoid-Enhanced CUR Decomposition LoRA, a novel PEFT variant designed to mitigate catastrophic forgetting. Our method introduces a novel normalization technique, Sigmoid-based Magnitude Norm (S-MagNorm), which enhances parameter retention and fine-tuning efficiency.
  arXiv  Detail & Related papers  (2025-02-25T13:00:05Z)
• LLM2: Let Large Language Models Harness System 2 Reasoning [65.89293674479907]
  Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We introduce LLM2, a novel framework that combines an LLM with a process-based verifier. LLMs2 is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs.
  arXiv  Detail & Related papers  (2024-12-29T06:32:36Z)
• Exploring Knowledge Boundaries in Large Language Models for Retrieval Judgment [56.87031484108484]
  Large Language Models (LLMs) are increasingly recognized for their practical applications. Retrieval-Augmented Generation (RAG) tackles this challenge and has shown a significant impact on LLMs. By minimizing retrieval requests that yield neutral or harmful results, we can effectively reduce both time and computational costs.
  arXiv  Detail & Related papers  (2024-11-09T15:12:28Z)
• LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints [86.59857711385833]
  We introduce RealInstruct, the first benchmark designed to evaluate LLMs' ability to follow real-world multi-constrained instructions. To address the performance gap between open-source and proprietary models, we propose the Decompose, Critique and Refine (DeCRIM) self-correction pipeline. Our results show that DeCRIM improves Mistral's performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
  arXiv  Detail & Related papers  (2024-10-09T01:25:10Z)
• STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions [6.19084217044276]
  Mitigating explicit and implicit biases in Large Language Models (LLMs) has become a critical focus in the field of natural language processing. We introduce the Sensitivity Testing on Offensive Progressions dataset, which includes 450 offensive progressions containing 2,700 unique sentences. Our findings reveal that even the best-performing models detect bias inconsistently, with success rates ranging from 19.3% to 69.8%.
  arXiv  Detail & Related papers  (2024-09-20T18:34:38Z)
• Navigating the Labyrinth: Evaluating and Enhancing LLMs' Ability to Reason About Search Problems [59.72548591120689]
  We introduce a new benchmark, SearchBench, containing 11 unique search problem types. We show that even the most advanced LLMs fail to solve these problems end-to-end in text. Instructing LLMs to generate code that solves the problem helps, but only slightly, e.g., GPT4's performance rises to 11.7%.
  arXiv  Detail & Related papers  (2024-06-18T00:44:58Z)
• Cutting Through the Noise: Boosting LLM Performance on Math Word Problems [52.99006895757801]
  Large Language Models excel at solving math word problems, but struggle with real-world problems containing irrelevant information. We propose a prompting framework that generates adversarial variants of MWPs by adding irrelevant variables. Fine-tuning on adversarial training instances improves performance on adversarial MWPs by 8%.
  arXiv  Detail & Related papers  (2024-05-30T18:07:13Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) can estimate causal effects under interventions on different parts of a system. We conduct empirical analyses to evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention. We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types, and enable a study of intervention-based reasoning.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• Evidence to Generate (E2G): A Single-agent Two-step Prompting for Context Grounded and Retrieval Augmented Reasoning [3.117335706912261]
  We introduce Evidence to Generate (E2G), a novel single-agent, two-step prompting framework. Instead of unverified reasoning claims, E2G focuses exclusively on the thought sequences explicitly mentioned in the context. tool achieves remarkable results robustly across a wide range of knowledge-intensive reasoning and generation tasks.
  arXiv  Detail & Related papers  (2024-01-11T09:49:15Z)
• Towards leveraging LLMs for Conditional QA [1.9649272351760063]
  This study delves into the capabilities and limitations of Large Language Models (LLMs) in the challenging domain of conditional question-answering. Our findings reveal that fine-tuned LLMs can surpass the state-of-the-art (SOTA) performance in some cases, even without fully encoding all input context. These models encounter challenges in extractive question answering, where they lag behind the SOTA by over 10 points, and in mitigating the risk of injecting false information.
  arXiv  Detail & Related papers  (2023-12-02T14:02:52Z)
• Cumulative Reasoning with Large Language Models [12.267474250936123]
  Cumulative Reasoning (CR) is a novel approach that utilizes language models cumulatively and iteratively. We demonstrate CR's superiority through several complex reasoning tasks. CR sets new state-of-the-art on the MATH dataset.
  arXiv  Detail & Related papers  (2023-08-08T16:18:20Z)
• TRAC: A Textual Benchmark for Reasoning about Actions and Change [7.79582487378263]
  Reasoning about actions and change (RAC) is essential to understand and interact with the ever-changing environment. With recent transformer-based language models (LMs), reasoning over text is desirable and seemingly feasible. We propose four essential RAC tasks as a comprehensive textual benchmark and generate problems in a way that minimizes the influence of other linguistic requirements.
  arXiv  Detail & Related papers  (2022-11-25T06:54:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.