DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

• URL: http://arxiv.org/abs/2606.12402v1
• Date: Wed, 10 Jun 2026 17:58:49 GMT
• Title: DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?
• Abstract summary: Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents.<n>We argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world.<n>We introduce DIRECT, a routing framework that uses multimodal scene context to allocate compute per prompt.
• Score: 57.585275546688116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents, with an emerging strategy of scaling test-time compute to improve capability. However, we observe that doing so increases latency, token usage, and FLOPs while yielding uneven, often diminishing gains in downstream success, limiting where embodied agents can be deployed. We argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world. We introduce DIRECT, a routing framework that uses multimodal scene context to allocate compute per prompt, improving the success--cost Pareto frontier over fixed model selection. Across three dominant scaling axes, namely chain-of-thought depth, model size, and memory history, our experiments on VLABench and RoboMME show that test-time compute is not a uniform lever: different axes yield qualitatively distinct capability gains. We validate these insights on a physical Franka arm in a DROID setup spanning zero-shot manipulation and long-horizon chaining, where our router matches or exceeds a stronger model's success rate at up to 65% lower average latency. Ultimately, our results show that naively scaling test-time compute is wasteful, and that DIRECT can provide frontier-level embodied planning in robotic systems at a fraction of the cost. Project page can be found at jadee-dao.github.io/direct/.

Related papers

• Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs [7.85315290601208]
  We present a systematic empirical study of inference-time scaling in local computer-use agents.<n>Our results show that additional computation often yields diminishing returns while changing failure modes.<n>Overall, our findings suggest that efficient local CUAs require selective compute allocation, failure-aware control mechanisms, and agentic frameworks.
  arXiv  Detail & Related papers  (2026-07-30T17:36:36Z)
• WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation [16.464426150252255]
  We present WARP (Warp-Augmented Relative Progress), a novel fully self-supervised algorithm for learning dense, signed relative progress magnitudes.<n> WARP generates per-frame progress targets via time-warp augmentations of demonstrations and we train WARP-RM to predict the normalized elapsed time between input frames.<n>We then introduce WARP-BC, which leverages these scalar reward estimates to upweight high-level action chunks during behavior cloning.
  arXiv  Detail & Related papers  (2026-06-26T17:58:06Z)
• Beyond Test-Time Training: Learning to Reason via Hardware-Efficient Optimal Control [86.63490309209378]
  We formulate reasoning as optimal control and introduce the Test-Time Control layer, which performs finite-horizon LQR planning over latent states at inference time.<n>We demonstrate that embedding optimal control as an architectural layer provides an effective and scalable mechanism for reasoning beyond test-time training.
  arXiv  Detail & Related papers  (2026-03-10T05:42:13Z)
• Agentic Test-Time Scaling for WebAgents [65.5178428849495]
  We present Confidence-Aware Test-Time Scaling (CATTS), which uses vote-derived uncertainty to allocate compute only when decisions are genuinely contentious.<n>CATTS improves performance on WebArena-Lite and GoBrowse by up to 9.1% over React while using up to 2.3x fewer tokens than uniform scaling.
  arXiv  Detail & Related papers  (2026-02-12T18:58:30Z)
• Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic [72.97800570813175]
  We propose Timely Machine, redefining test-time as wall-clock time.<n>We introduce Timely-Eval, a benchmark spanning high-frequency tool calls, low-frequency tool calls, and time-constrained reasoning.<n>We find smaller models excel with fast feedback through more interactions, while larger models dominate high-latency settings via superior interaction quality.
  arXiv  Detail & Related papers  (2026-01-23T06:28:52Z)
• Hidden States as Early Signals: Step-level Trace Evaluation and Pruning for Efficient Test-Time Scaling [19.080366193748127]
  Large Language Models (LLMs) can enhance reasoning capabilities through test-time scaling by generating multiple traces.<n>The combination of lengthy reasoning traces with multiple sampling introduces substantial computation and high end-to-end latency.<n>We propose STEP: Step-level Trace Evaluation and Pruning, a novel pruning framework that evaluates reasoning steps using hidden states.
  arXiv  Detail & Related papers  (2026-01-14T02:54:55Z)
• Exploring Test-time Scaling via Prediction Merging on Large-Scale Recommendation [13.057539100440634]
  How to efficiently utilize and scale up computational resources during test time remains underexplored.<n>Key point in applying test-time scaling to DLRS lies in effectively generating diverse yet meaningful outputs.<n>Test-time scaling can be seamlessly accelerated with the increase in parallel servers when deployed online.
  arXiv  Detail & Related papers  (2025-12-08T15:41:10Z)
• Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models [52.502867924372275]
  Mixture-of-Experts (MoE) models achieve efficient scaling through sparse expert activation, but often suffer from suboptimal routing decisions due to distribution shifts in deployment.<n>We propose textita data-free, online test-time framework that continuously adapts MoE routing decisions during text generation without external supervision or data.
  arXiv  Detail & Related papers  (2025-10-16T16:24:36Z)
• $\texttt{SPECS}$: Faster Test-Time Scaling through Speculative Drafts [55.231201692232894]
  $textttSPECS$ is a latency-aware test-time scaling method inspired by speculative decoding.<n>Our results show that $textttSPECS$matches or surpasses beam search accuracy while reducing latency by up to $sim$19.1%.
  arXiv  Detail & Related papers  (2025-06-15T05:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.