ScratchWorld: Evaluating If World Models Compute Executable Consequences

• URL: http://arxiv.org/abs/2606.31689v1
• Date: Tue, 30 Jun 2026 14:02:25 GMT
• Title: ScratchWorld: Evaluating If World Models Compute Executable Consequences
• Abstract summary: We introduce ScratchWorld, an offline diagnostic benchmark that treats Scratch projects as executable worlds.<n> ScratchWorld evaluates next-state prediction, long-horizon tracking, causal event attribution, and counterfactual prediction.
• Score: 2.3981743514438087
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World-model evaluations often score a predicted future by overlap with a target state or observation. In sparse-change worlds, this can turn copied persistent state into apparent accuracy. We introduce ScratchWorld, an offline diagnostic benchmark that treats Scratch projects as executable worlds and uses a pinned Scratch VM to produce replay-verified transitions, hidden variables, causal traces, and counterfactual outcomes. ScratchWorld evaluates next-state prediction, long-horizon tracking, causal event attribution, and counterfactual prediction; each replay-verified target can be presented under raw-program, structured-state, natural-language, or rendered input modalities, and our experiments use the structured-state condition. Its primary state metric is value-aware changed-field $F_1$, which gives credit only for the changed field and its executed value. In a 659-example release, seven prompted language/reasoning models reach at most 13.8% value-aware changed-field $F_1$ in a state-only partial-observation stress test. A same-instance copy diagnostic makes the overlap confound concrete: copying the input state reaches 98.0% implied full-state field accuracy and 0.0% changed-field $F_1$, with the largest inflation on real projects. Auxiliary diagnostics separate hidden-state rollout drift, intervention sensitivity, causal attribution, and perturbation robustness. Across these settings, models often react to actions or interventions without following the executable rule that determines the changed value.

Related papers

• Confident at the moment of action: belief miscalibration in LLM play under hidden information [0.0]
  Agentic systems increasingly gate actions on a model's own stated confidence, which assumes confidence tracks correctness at the moment of acting.<n>We test this in a hidden-information chess variant where royal status can be secretly, repeatedly relocated between pieces.<n>A model exhibiting this pattern can still win the game its belief was about, which is why outcome-only evaluation would not detect it.
  arXiv  Detail & Related papers  (2026-08-25T15:18:18Z)
• Reading the Room: Implicit Confusion Encoding in Recurrent World Model States [0.0]
  We show a recurrent hidden state $h_t$ trained to reduce prediction error tracks its own confusion.<n>It is functionally distinct from ensemble disagreement, which flags new inputs, and reconstruction error, which flags bad predictions right now.
  arXiv  Detail & Related papers  (2026-08-21T19:32:38Z)
• The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping [70.07961857926377]
  We introduce trace-grounded parametric profiling for event counting in three controlled video tasks.<n>Across 2,190 videos, we vary event count N and frequency F while holding rendering fixed.<n>At an 80% reliability threshold, Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5 and 1.0 Hz, yet demonstrates no reliable positive-count region for transient blinking events.
  arXiv  Detail & Related papers  (2026-08-06T17:57:06Z)
• Exposure Bias as Epistemic Underidentification in Recursive Forecasting [1.2489632787815885]
  We show that, under partial observability or state truncation, recursive rollout is also an underidentification problem.<n>Even with deterministic latent dynamics, one-step Bayes supervision identifies behavior only on observed contexts.<n>We formalize this with induced states $Z$ and provenance variables $P$, and derive a decomposition of induced-state error into teacher-forcing/rollout mismatch.
  arXiv  Detail & Related papers  (2026-06-11T07:25:15Z)
• PatchWorld: Gradient-Free Optimization of Executable World Models [69.05094454437402]
  We introduce PatchWorld, a gradient-free framework that turns offline trajectories into executable Python world models.<n>Across seven AgentGym environments, PatchWorld-Simple achieves the highest code-based planning score.<n>We find that a human-specified residual-memory bias improves surface observation fidelity but weakens decision utility.
  arXiv  Detail & Related papers  (2026-05-29T06:11:14Z)
• Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences [2.7214777196418645]
  We introduce Chess-World-Model, a large-scale state-tracking benchmark built from 10 million real chess games.<n>We benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues.<n>Real-game performance saturates above 18 million parameters, but the random-uniform split remains discriminative up to 40 million, exposing failures otherwise hidden by scale.
  arXiv  Detail & Related papers  (2026-05-28T15:43:31Z)
• World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models [3.1727619150610837]
  Vision-language models (VLMs) are increasingly used to answer questions about physical scenes.<n>We introduce wmw, an evaluation framework for auditing the emphlanguage-expressed physical commitments ofVLMs.<n>A hybrid verifier then checks schema validity, state grounding, transition consistency, and answer-trace compatibility.
  arXiv  Detail & Related papers  (2026-05-28T08:29:32Z)
• Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation [98.79811866787263]
  We introduce MetaFine, a diagnostic meta-evaluation framework that disentangles manipulation competency along three axes.<n>We identify the visual encoder's ability to preserve local spatial structure as a key bottleneck for fine-grained precision.<n>By shifting evaluation from ranking to diagnosis, MetaFine turns benchmarking into an actionable compass for repairing the layered capacities underlying genuine physical dexterity.
  arXiv  Detail & Related papers  (2026-05-19T15:25:13Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models [0.7919969809015935]
  We show that models that maintain temporally grounded beliefs generalize better.<n>We formalize this as behavioral faithfulness over long horizons.<n>We show that temporal grounding quality is a leading indicator of robustness.
  arXiv  Detail & Related papers  (2026-03-06T19:43:26Z)
• UncTrack: Reliable Visual Object Tracking with Uncertainty-Aware Prototype Memory Network [75.9933952886197]
  UncTrack is a novel uncertainty-aware transformer tracker that predicts the target localization uncertainty.<n>Our method outperforms other state-of-the-art methods in experiments.
  arXiv  Detail & Related papers  (2025-03-17T07:33:16Z)
• OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking [55.62705574507595]
  OpenPI is the only dataset annotated for open-vocabulary state tracking. We categorize 3 types of problems on the procedure level, step level and state change level respectively. For the evaluation metric, we propose a cluster-based metric to fix the original metric's preference for repetition.
  arXiv  Detail & Related papers  (2023-06-01T16:48:20Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [49.15931834209624]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world.<n>We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.<n>By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.