StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2608.26067v1
• Date: Wed, 26 Aug 2026 17:33:19 GMT
• Title: StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models
• Abstract summary: StreamPI is a streaming multimodal temporal modeling framework.<n>It equips single-frame VLA with temporal reasoning capability without introducing any additional parameters.<n> Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
• Score: 84.17128030384099
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.

Related papers

• COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models [15.39037309359101]
  Video multimodal large language models have advanced significantly, yet fine-grained motion-temporal understanding remains fragile.<n>We propose COMET, a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation.<n>For optimization, COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage that turns temporal order into a direct learning signal.
  arXiv  Detail & Related papers  (2026-08-21T12:28:36Z)
• TBD-VLA: Temporal Block Diffusion Vision Language Action Model [7.861095039299131]
  We introduce TBD-VLA, a discrete token-based VLA framework that incorporates block diffusion to enable temporal action generation.<n>We partition action sequences into temporal blocks and perform masked discrete diffusion within each block, while maintaining autoregressive generation across blocks.<n>This design unifies temporal autoregression and parallel action decoding, achieving both strong temporal coherence and improved inference speed.
  arXiv  Detail & Related papers  (2026-06-05T23:10:43Z)
• MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction [76.4461698685681]
  Mini-o 4.5 is our latest effort towards human-level realtime streaming interaction.<n>It is based on Omni-CPM, a unified streaming framework that aligns omni-modal perception and outputs along a shared temporal axis.<n>With a total of 9B parameters, Mini-o 4.5 approaches Gemini 2.5 Flash in vision computation capabilities, delivering stateofthe-art open computation performance scale.
  arXiv  Detail & Related papers  (2026-04-30T04:05:43Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• VLA-IAP: Training-Free Visual Token Pruning via Interaction Alignment for Vision-Language-Action Models [27.12266806191131]
  Vision-Language-Action (VLA) models have rapidly advanced embodied intelligence, enabling robots to execute complex, instruction-driven tasks.<n>Current approaches often prune visually sparse yet structurally critical regions that support manipulation, leading to unstable behavior during early task phases.<n>Our proposed textbftraining-free method, VLA-IAP (Interaction-Aligned Pruning), introduces a geometric prior mechanism to preserve structural anchors and a dynamic scheduling strategy.
  arXiv  Detail & Related papers  (2026-03-24T09:33:05Z)
• dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thought [66.78110237549087]
  Vision-Language-Action (VLA) models are emerging as a next-generation paradigm for robotics.<n>We introduce dVLA, a diffusion-based VLA that unifies visual perception, language reasoning, and robotic control in a single system.
  arXiv  Detail & Related papers  (2025-09-30T02:36:11Z)
• CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modeling [84.51372201195132]
  CronusVLA is a unified framework that extends single-frame VLA models to the multi-frame paradigm.<n>CronusVLA achieves leading performance and superior robustness, with a 70.9% success rate.<n>These results highlight the potential of efficient multi-frame adaptation in VLA models for more powerful and robust real-world deployment.
  arXiv  Detail & Related papers  (2025-06-24T17:30:27Z)
• MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis [32.08769443927576]
  We propose Manipulate in Dream (MinD), a dual-system world model for real-time, risk-aware planning.<n>MinD uses two asynchronous diffusion processes: a low-frequency visual generator (LoDiff) that predicts future scenes and a high-frequency diffusion policy (HiDiff) that outputs actions.<n>Our key insight is that robotic policies do not require fully denoised frames but can rely on low-resolution latents generated in a single denoising step.<n>MinD achieves a 63% success rate on RL-Bench, 60% on real-world Franka tasks, and operates at 11.3 FPS
  arXiv  Detail & Related papers  (2025-06-23T17:59:06Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.