The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom

• URL: http://arxiv.org/abs/2608.10145v1
• Date: Mon, 10 Aug 2026 19:00:51 GMT
• Title: The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom
• Abstract summary: LeModel trains a latent world model with a prediction loss and a single anticollapse regulariser.<n>We reproduce that by independent re-implementation on roughly $25 of rented compute, with all evaluation on one laptop CPU.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LeWorldModel trains a latent world model with a prediction loss and a single anti-collapse regulariser, and reports approximately 87% of goals reached on TwoRoom, its simplest diagnostic environment. We reproduce that result by independent reimplementation on roughly $25 of rented compute, with all evaluation on one laptop CPU. We reach 94.0% at the repository's evaluation goal offset, against 84.0% for the authors' own released checkpoint measured under our protocol on identical episodes, and we reproduce the reported representation result directly (position probe Pearson r = 0.9988 against a reported 0.996). Reaching that point required four conventions that determine the outcome and appear in no released configuration file: dense action gathering across a frameskip block, a programmatically-set action-encoder width, ImageNet pixel normalisation, and action z-scoring. A reproducer following the released configurations alone obtains a model whose predictor cannot converge. The evaluation protocol is itself contested by the released material. The paper's appendix and the repository's configuration specify different goal offsets and step budgets; on the authors' own weights these yield 14.0% and 84.0%, and only the configuration's values reproduce the reported figure. On fifty identical episodes, changing nothing but how the goal is constructed moves that checkpoint from 84.0% to 8.0%. Two findings generalise. One-step prediction accuracy does not predict long-horizon planning success: across three checkpoints spanning a sevenfold range in prediction error, including the authors' own, it orders short-horizon success monotonically and fails to order long-horizon success at all. And a batch normalisation layer inflated our reported validation loss by up to a factor of 300, concealing a training loss that was flat throughout.

Related papers

• ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
  Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
  arXiv  Detail & Related papers  (2026-09-15T04:59:03Z)
• Modeling What Changes: Sparse, Residual World Models for Object-Centric Manipulation [0.0]
  We show that the sparse/residual model predicts next-state poses 2.5 to 4.6 times more accurately than a dense multilayer perceptron.<n>In autoregressive rollout it compounds far less error, hugging the no-motion floor while the dense model drifts.<n> Modeling what changes, rather than re-predicting the whole world, is a simple, effective bias for object-centric physical AI.
  arXiv  Detail & Related papers  (2026-09-02T03:20:09Z)
• Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair [0.0]
  Fault localization can focus a code model's repair on the statements a failing test implicates, but a targeted edit may succeed merely because it is small.<n>We separate these explanations with three arms applied to the same failed candidate.<n>Across three frozen 26-32B models, three benchmarks and 488 failing candidates, three results follow.
  arXiv  Detail & Related papers  (2026-09-01T07:48:51Z)
• A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction [45.496856835431885]
  3D reconstruction models emit per-pixel confidence that downstream systems read as a reliability signal.<n>We score the confidence on four properties, how well it ranks error, whether its level is right on average, whether it holds across the confidence range, and whether its intervals cover the truth.<n>The confidence ranks error well, but the predicted uncertainty is too low when it is read under conditions that are not exactly those of training.
  arXiv  Detail & Related papers  (2026-08-30T10:21:34Z)
• When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse [3.0255780640228984]
  Joint-embedding predictive architectures are selected almost universally by linear probing and effective rank.<n>We report a case where both read healthily while the representation carries zero usable instance information.<n>A Graph-JEPA predicts one masked aspect from a subgraph's remaining aspects, attaining linear-probe accuracy 0.871 and effective rank 18-47.<n>We release a harness with a reducibility audit and target gate.
  arXiv  Detail & Related papers  (2026-08-20T19:22:49Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• One Ruler: A Same-Hands Re-Evaluation of Bivariate Causal Direction on Tuebingen, with a Parameter-Free Compression Baseline [0.0]
  We re-evaluate the Tuebingen cause-effect pairs in which every method is run by us on the identical 102 pairs.<n>As a clean reference point we introduce a deliberately minimal baseline.<n>Under the common ruler the ranking differs sharply from the literature.
  arXiv  Detail & Related papers  (2026-06-22T15:43:19Z)
• Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness [45.148328075418156]
  A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
  arXiv  Detail & Related papers  (2026-06-10T05:55:13Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents [11.233308795768465]
  VIGIL is an evaluation framework that makes terminal commitment independently measurable.<n>Under VIGIL's default protocol, agents observe only egocentric RGB, receive no action-success signals, and must end each episode with a semantic report checked deterministically against hidden world state.<n>This yields two separate scores: world-state completion (W) and benchmark success (B)
  arXiv  Detail & Related papers  (2026-05-09T07:24:07Z)
• When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models [2.064923532131528]
  Deployed language models must produce outputs that are both correct and format-compliant.<n>We study this structured-output reliability gap using two mathematical benchmarks -- GSM8K and MATH.<n>We develop AloLab, an iterative system-prompt requiring only black-box API access to the target model.
  arXiv  Detail & Related papers  (2026-05-04T09:07:44Z)
• vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models [58.633451339058986]
  VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository.<n>We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution.<n>A complete evaluation requires only two commands: vla eval serve and vla eval run.
  arXiv  Detail & Related papers  (2026-03-14T14:38:53Z)
• Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making [38.75183725659772]
  Large language models (LLMs) are increasingly used for tasks that require complex reasoning.<n>We argue that measuring internal processes is essential for understanding model behavior and improving reliability.<n>We introduce a framework that evaluates LLMs along three core dimensions: planning, revision, and resource-constrained decision making.
  arXiv  Detail & Related papers  (2025-06-13T17:59:10Z)
• Confidence Adaptive Anytime Pixel-Level Recognition [86.75784498879354]
  Anytime inference requires a model to make a progression of predictions which might be halted at any time. We propose the first unified and end-to-end model approach for anytime pixel-level recognition.
  arXiv  Detail & Related papers  (2021-04-01T20:01:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.