Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs

• URL: http://arxiv.org/abs/2606.29350v1
• Date: Sun, 28 Jun 2026 11:42:55 GMT
• Title: Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs
• Abstract summary: Vision-language models and vision-language action models endow the robot with unprecedented capabilities.<n>The input of video and high-resolution images yields a massive number of visual tokens, leading to extremely high inference latency and severely hindering the robot's real-time control.<n>We propose ST-Merge, a plug-and-play framework that efficiently fuses redundant tokens directly during the visual encoding phase.
• Score: 21.02858354959528
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-language models and vision-language action models endow the robot with unprecedented capabilities. However, the input of video and high-resolution images yields a massive number of visual tokens, leading to extremely high inference latency and severely hindering the robot's real-time control. To break through this computational bottleneck, we propose ST-Merge, a plug-and-play, training-free framework that efficiently fuses redundant tokens directly during the visual encoding phase. By explicitly constructing 3D spatiotemporal coordinates, it employs a multi-queue parallel matching and weighted aggregation mechanism to achieve efficient and geometrically consistent fusion of redundant tokens across frames. In addition, we introduce a post-merge positional correction mechanism that effectively eliminates spatial deviation caused by merging by dynamically re-evaluating the rotational position code of the weighted centroid of the vision token, thereby ensuring the high-precision spatial awareness required for dexterous operation. In the Video Question Answering task on the mainstream VLM, Qwen2.5-VL, ST-Merge achieves a 2$\times$ inference speedup with only a tiny 1\% loss in precision. When deployed on the $π_{0.5}$ VLA policy, ST-Merge achieves an 8.3$\times$ speedup at 1024 $\times$ 1024 resolution and matches the baseline success rate at this high-resolution setting. At lower resolutions, it introduces a small drop in accuracy.

Related papers

• Reducing Temporal Redundancy for Efficient Vision-Language-Action Inference [19.683948028719808]
  Vision-Language-Action (VLA) models exhibit strong generalization for robotic manipulation, yet their high inference latency limits real time deployment.<n>We propose a system level acceleration strategy that reduces computation in both perception and action generation.<n> Experiments on Libero, RobotWin, and Real Robot Platforms demonstrate over 2 times speedup while maintaining high performance.
  arXiv  Detail & Related papers  (2026-07-14T02:48:31Z)
• Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing [55.13802890769086]
  We present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing.<n>To achieve profound modal synergy, Evita embeds a suite of intrinsic co-learning modules directly into every encoder layer.<n>Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.
  arXiv  Detail & Related papers  (2026-07-10T06:52:09Z)
• ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation [6.976481623762446]
  textttReactVLA is a lightweight and low-latency VLA framework for real-time robotic manipulation.<n>textttReactVLA consistently outperforms similarly sized VLA baselines.<n>It reduces real-world policy latency to below 38.6 ms, enabling fast reactive control on physical robot platforms.
  arXiv  Detail & Related papers  (2026-06-12T08:33:37Z)
• REALM: Retrospective Encoder Alignment for LFP Modeling [2.8036525859094823]
  Spike activity has been the dominant neural signal for behavior decoding due to its high spatial and temporal resolution.<n>As brain-computer interfaces (BCIs) move toward high channel counts and wireless operation, the high sampling frequency of spike signals becomes a bottleneck.<n>We propose REALM: a retrospective distillation framework that enables causal LFP decoding.
  arXiv  Detail & Related papers  (2026-05-14T14:16:22Z)
• DA-PTQ: Drift-Aware Post-Training Quantization for Efficient Vision-Language-Action Models [61.26694413585136]
  Drift-Aware Post-Training Quantization (DA-PTQ) formulates quantization as a drift-aware optimization problem over sequential decision processes.<n>DA-PTQ significantly reduces kinematic drift and achieves comparable performance to full-precision models under low-bit settings.
  arXiv  Detail & Related papers  (2026-04-13T14:51:43Z)
• DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks [54.32016216994156]
  We introduce the Causal Latent World Model (CLWM), which employs DINOv3 features as generative targets to disentangle interaction semantics from visual noise.<n>To overcome memory scaling, CLWM features a Dual-State Test-Time Training (TTT) Memory that guarantees a strict $mathcalO(1)$ footprint for long-horizon tasks.<n>We present EmbodiChain, an online framework that establishes the Efficiency Law by injecting an infinite flow of physics-grounded trajectories during training.
  arXiv  Detail & Related papers  (2026-04-13T03:19:36Z)
• Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models [41.32939197803798]
  Q-Zoom is a query-aware adaptive high-resolution perception framework.<n>A lightweight Gating Network safely bypasses high-resolution processing when coarse global features suffice.<n>A Self-Distilled Region Network precisely localizes the task-relevant Region-of-Interest from intermediate feature spaces.
  arXiv  Detail & Related papers  (2026-04-08T10:12:30Z)
• $D^3$-RSMDE: 40$\times$ Faster and High-Fidelity Remote Sensing Monocular Depth Estimation [72.9912717963138]
  Real-time, high-fidelity monocular depth estimation from remote sensing imagery is crucial for numerous applications.<n>Although using Vision Transformer (ViT) backbones for dense prediction is fast, they often exhibit poor perceptual quality.<n>We propose Depth Detail Diffusion for Remote Sensing Monocular Depth Estimation ($D3$-RSMDE) to overcome these limitations.<n>$D3$-RSMDE achieves a notable 11.85% reduction in the Learned Perceptual Image Patch Similarity (LPIPS) perceptual metric.
  arXiv  Detail & Related papers  (2026-03-17T10:50:36Z)
• Towards Lossless Ultimate Vision Token Compression for VLMs [11.485425012979052]
  Lossless Ultimate Vision tokens Compression (LUVC) framework is proposed.<n>LUVC compresses visual tokens until complete elimination at the final layer of language model.<n>Experiments show that LUVC achieves a 2 speedup inference in language model with negligible accuracy degradation.
  arXiv  Detail & Related papers  (2025-12-09T15:40:13Z)
• H$_{2}$OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers [124.11648300910444]
  We present a hierarchical plug-and-play pruning-and-$-recovering framework, called Hierarchical Hourglass Tokenizer (H$_2$OT)<n>Our method is general-purpose: it can be easily incorporated into common VPT models on both seq2seq and seq2frame pipelines.
  arXiv  Detail & Related papers  (2025-09-08T17:59:59Z)
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning [80.104336426172]
  Chain-of-thought (CoT) enhances problem-solving ability of large language models.<n>CoT incurs substantial inference cost due to long autoregressive trajectories.<n>We introduce R-Stitch, a training-free hybrid decoding framework.
  arXiv  Detail & Related papers  (2025-07-23T08:14:36Z)
• SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration [70.72227437717467]
  Vision-Language-Action (VLA) models have attracted increasing attention for their strong control capabilities.<n>Their high computational cost and low execution frequency hinder their suitability for real-time tasks such as robotic manipulation and autonomous navigation.<n>We propose SP-VLA, a unified framework that accelerates VLA models by jointly scheduling models and pruning tokens.
  arXiv  Detail & Related papers  (2025-06-15T05:04:17Z)
• FastVLM: Efficient Vision Encoding for Vision Language Models [22.41836943083826]
  We introduce FastVLM, a model that achieves an optimized trade-off between latency, model size and accuracy.<n>FastVLM incorporates FastViTHD, a novel hybrid vision encoder designed to output fewer tokens.<n>In the LLaVA-1.5 setup, FastVLM achieves 3.2$times$ improvement in time-to-first-token.
  arXiv  Detail & Related papers  (2024-12-17T20:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.