DeepInsight II: One Trace from Benchmark to Robot
- URL: http://arxiv.org/abs/2608.16556v1
- Date: Mon, 17 Aug 2026 13:25:24 GMT
- Title: DeepInsight II: One Trace from Benchmark to Robot
- Abstract summary: DeepInsight II keeps that substrate fixed and quantifies the embodied half.<n> MotionBench places four released whole-body controllers under one workload and metric contract.<n>A composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels.
- Score: 8.388561928330978
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while the embodied layers on which deployment actually turns remain fragmented across benchmark-specific simulators, embodiments, and interfaces. The first DeepInsight report (v1) unified evaluation across this stack behind three abstractions---task, resource, and result---but its quantitative evidence centered on the foundation-model layer; navigation and manipulation (System 1) and whole-body control (System 0) remained simulation case studies, and physical execution was outside its empirical scope. DeepInsight II keeps that substrate fixed and quantifies the embodied half. First, it reproduces released-checkpoint references across two navigation and four manipulation benchmarks under their native protocols. Second, MotionBench places four released whole-body controllers under one workload and metric contract, then carries a qualified within-family cohort from parallel simulation to matched real-robot trials in which simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific records, making the sim-to-real gap a native reduction rather than a reconciliation across toolchains. Third, a composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels, each mapped to a concrete repair action, with a measured repairability criterion and physical episodes testing the same attribution under hardware-observable state. The contribution is therefore not a new evaluation architecture, but empirical continuity from benchmark execution to matched robot evidence and repair-oriented diagnosis.
Related papers
- Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
arXiv Detail & Related papers (2026-08-17T09:53:27Z) - Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
arXiv Detail & Related papers (2026-08-11T18:48:40Z) - Compositional Benchmark Synthesis for Hierarchical Human Action Recognition [4.812022454180755]
A benchmark-generation and evaluation frameworkis proposed that synthesizes a four-level hierarchical-intention benchmark.<n> Episodes are assembled by a transition model under a subject-consistency constraint.<n>A coverage-aware sampler reduces the subject usage Gini from 0.566 to 0.248.
arXiv Detail & Related papers (2026-08-11T10:22:22Z) - MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents [7.0922719251235655]
MetaSpace is a novel framework designed to evaluate the spatial cognition of embodied agents.<n>We show that MetaSpace successfully detects 90,422 spatial cognition errors in state-of-the-art (SOTA) MLLM-driven agents.<n>Results indicate that all SOTA agents achieve average scores between 0.44 and 0.52, significantly lower than the human benchmark of 0.96.
arXiv Detail & Related papers (2026-07-26T14:07:20Z) - PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
arXiv Detail & Related papers (2026-07-18T04:46:53Z) - DeepInsight: A Unified Evaluation Infrastructure Across the Physical AI Stack [17.770542038652568]
evaluate a Physical AI stack spans operators that differ by more than three orders of magnitude.<n>No existing framework spans this range, so the stack is evaluated today by stitching together separate harnesses.<n>We present DeepInsight, an evaluation infrastructure that serves this full spectrum on a single runtime.
arXiv Detail & Related papers (2026-06-16T06:22:09Z) - Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation [98.79811866787263]
We introduce MetaFine, a diagnostic meta-evaluation framework that disentangles manipulation competency along three axes.<n>We identify the visual encoder's ability to preserve local spatial structure as a key bottleneck for fine-grained precision.<n>By shifting evaluation from ranking to diagnosis, MetaFine turns benchmarking into an actionable compass for repairing the layered capacities underlying genuine physical dexterity.
arXiv Detail & Related papers (2026-05-19T15:25:13Z) - Computer Use at the Edge of the Statistical Precipice [19.604476051265852]
We show that a 1MB replay script that blindly executes a recorded action sequence outperforms frontier models on prominent static benchmarks.<n>We trace failures to two root causes: non-principled environment design and non-principled evaluation methodology.
arXiv Detail & Related papers (2026-05-07T22:43:43Z) - Dynamic analysis enhances issue resolution [53.50448142467294]
DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
arXiv Detail & Related papers (2026-03-23T14:48:54Z) - Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
arXiv Detail & Related papers (2026-03-13T20:44:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.