ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

• URL: http://arxiv.org/abs/2605.24305v1
• Date: Sat, 23 May 2026 00:34:17 GMT
• Title: ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
• Abstract summary: We present ChaosBench-Logic v2, a 40,886-question benchmark over 165 dynamical systems.<n>We find that regime-transition reasoning remains near random even for frontier models.<n>Per-family decomposition shows that the proprietary-model advantage concentrates on cross-indicator and consistency tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard accuracy on binary reasoning benchmarks hides critical failure modes: prior collapse, inconsistency under paraphrase, and inability to reason about parameter-dependent dynamics. We present ChaosBench-Logic v2, a 40,886-question benchmark over 165 dynamical systems with 27 FOL predicates and 78 axiom edges, together with CARE (Calibration- and Adversarial-Robust Evaluation), a protocol that surfaces these pathologies. Evaluating 14 models, we find that regime-transition reasoning remains near random (MCC = 0.05) even for frontier models, whereas FOL deduction with given premises reaches MCC = 0.52. Per-family decomposition shows that the proprietary-model advantage concentrates on cross-indicator (+0.40) and consistency tasks, while open-source Qwen 2.5-32B dominates indicator diagnostics (0.91 vs. 0.45). Two models exhibit negative MCC on bifurcation questions, confirmed as systematic anti-correlation via confusion-matrix analysis.

Related papers

• Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment [6.781586922950944]
  We introduce LogiMed-RoB, a benchmark grounded in Cochrane Risk of Bias (RoB) 2.0 expert logic.<n>It evaluates models under the Hierarchical Logical Consistency framework across four dimensions.<n> Experiments on 10 state-of-the-art LLMs reveal a catastrophic Error Compounding Effect.
  arXiv  Detail & Related papers  (2026-09-10T07:46:41Z)
• FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses [0.0]
  FailureScope is a behavioral-diagnosis method that clusters evaluation probes by their cross-model pass/fail patterns.<n>We show it yields stable, interpretable failure across three regimes usually studied separately: single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks.
  arXiv  Detail & Related papers  (2026-06-03T01:28:00Z)
• SL-BiLEM: Structured Learnable Behavior-in-the-Loop Epidemic Modeling for Forecasting and Policy Evaluation [42.644852156101685]
  Epidemic forecasting faces a fundamental challenge: human behavior dynamically responds to disease spread.<n>We propose textbfSL-BiLEM (Structured Learnable Behavior-in-the-Loop Epidemic Model), leveraging physical constraints as regularization for robust extrapolation.
  arXiv  Detail & Related papers  (2026-05-26T08:46:04Z)
• Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning [56.48520300004217]
  We present Seirnes, a self-play RL framework that transforms contextual interference into an internal training signal.<n>A single model is trained to both construct plausible yet distracting contexts that expose its own reasoning blind spots.<n>By pitting these competing objectives against each other, Seirnes compels the model to move beyond superficial pattern matching.
  arXiv  Detail & Related papers  (2026-05-12T06:58:35Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• Robust Reasoning Benchmark [2.622879924644665]
  We propose a perturbation pipeline consisting of 14 techniques to evaluate robustness of LLM reasoning.<n>We apply this pipeline to AIME 2024 dataset and evalute 8 state-of-the-art models on the resulting benchmark.<n>Our results indicate that open weight models ranging from 7B to 120B parameters and Claude Opus 4.6 exhibit accuracy decay on subsequent problems.
  arXiv  Detail & Related papers  (2026-03-26T22:19:33Z)
• A Unified Framework for Joint Detection of Lacunes and Enlarged Perivascular Spaces [3.9313804276175506]
  Cerebral small vessel disease (CSVD) markers, specifically enlarged perivascular spaces (EPVS) and lacunae, present a unique challenge in medical image analysis.<n>We propose a morphologydecoupled framework where Zero- Gated CrossTask Attention exploits dense EPVS context to guide sparse lacune detection.
  arXiv  Detail & Related papers  (2026-03-04T16:30:46Z)
• Evaluating and Enhancing the Vulnerability Reasoning Capabilities of Large Language Models [15.849480549367684]
  We propose DAGVul, a novel framework that models vulnerability reasoning as a Directed Acyclic Graph (DAG) generation task.<n>By further introducing Reinforcement Learning with Verifiable Rewards (RLVR), we align model reasoning trace with program-intrinsic logic.<n>Our framework improves the reasoning F1-score by an average of 18.9% over all the baselines.
  arXiv  Detail & Related papers  (2026-02-06T13:19:45Z)
• Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification [49.506412445511934]
  Large Language Models (LLMs) show remarkable capabilities, yet their next-token prediction creates logical inconsistencies and reward hacking.<n>We introduce a formal logic verification-guided framework that dynamically interleaves formal symbolic verification with the natural language generation process.<n>We operationalize this framework via a novel two-stage training pipeline that synergizes formal logic verification-guided supervised fine-tuning and policy optimization.
  arXiv  Detail & Related papers  (2026-01-30T07:01:25Z)
• ReLE: A Scalable System and Structured Benchmark for Diagnosing Capability Anisotropy in Chinese LLMs [37.23311145049677]
  We present ReLE, a scalable system designed to diagnose Capability Anisotropy.<n>We evaluate 304 models across a Domain $times$ Capability Symbolic matrix comprising 207,843 samples.
  arXiv  Detail & Related papers  (2026-01-24T09:57:59Z)
• Prediction and Causality of functional MRI and synthetic signal using a Zero-Shot Time-Series Foundation Model [46.186152268413025]
  We evaluate a foundation model against classical methods for inferring directional interactions from brain activity measured with fMRI in humans.<n>We tested the forecasting ability of the foundation model in both zero-shot and fine-tuned settings, and assessed causality by comparing Granger-like estimates from the model with standard Granger causality.
  arXiv  Detail & Related papers  (2025-09-15T22:43:23Z)
• Organ-Agents: Virtual Human Physiology Simulator via LLMs [66.40796430669158]
  Organ-Agents is a multi-agent framework that simulates human physiology via LLM-driven agents.<n>We curated data from 7,134 sepsis patients and 7,895 controls, generating high-resolution trajectories across 9 systems and 125 variables.<n>Organ-Agents achieved high simulation accuracy on 4,509 held-out patients, with per-system MSEs 0.16 and robustness across SOFA-based severity strata.
  arXiv  Detail & Related papers  (2025-08-20T01:58:45Z)
• CRTRE: Causal Rule Generation with Target Trial Emulation Framework [47.2836994469923]
  We introduce a novel method called causal rule generation with target trial emulation framework (CRTRE) CRTRE applies randomize trial design principles to estimate the causal effect of association rules. We then incorporate such association rules for the downstream applications such as prediction of disease onsets.
  arXiv  Detail & Related papers  (2024-11-10T02:40:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.