vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.08094v1
• Date: Sat, 06 Jun 2026 10:45:40 GMT
• Title: vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
• Abstract summary: We present vla, a portable C++ inference runtime built on llama.<n>It is the first ggml-class engine to serve the flow-matching and diffusion VLA inference pattern.<n>On LIBERO-Object, the engine matches a state-of-the-art checkpoint within one episode out of 200, and runs BitVLA at 100% success in 1.3 GiB of memory.
• Score: 1.8512656289778031
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) policies are typically shipped as Python/PyTorch stacks that assume a workstation-class GPU, a mismatch for the hardware on which robots actually run. We present vla.cpp, a portable C++ inference runtime built on llama.cpp. To our knowledge, it is the first ggml-class engine to natively serve the flow-matching and diffusion VLA inference pattern, in which a cached vision-language prefix is consumed by a cross-attending action expert integrated over several solver steps. A single runtime serves seven architectures spanning five backbone and four action-head families behind one request/response protocol, with each model packaged as a self-contained bundle. On LIBERO-Object, the engine matches a state-of-the-art checkpoint to within one episode out of 200, and runs BitVLA at 100% success in 1.3 GiB of memory. The same bundle runs unchanged across three hardware tiers, from a consumer GPU down to an 8 GB embedded module. A cross-hardware roofline analysis shows that batch-1 VLA inference is compute-bound, so utilization rather than bandwidth is the deployment lever; an IMMA ladder GEMM derived from this analysis cuts BitVLA per-step latency by 4.5x. We then frame an on-robot stress test on an ALOHA arm that isolates the latency constraint under which a learned VLA must replan against a moving target on the hardware it was trained for. Code, demo videos, and the reproducible benchmark scaffold are available at https://fai-modelopt-tech.github.io/vla-cpp.github.io/.

Related papers

• Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets [0.0]
  We show that a handful of AIPCs can serve models beyond the capability of any single one.<n>A four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed.
  arXiv  Detail & Related papers  (2026-08-19T17:33:28Z)
• Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots [19.848719701819153]
  We present Embodied$.$, a portable runtime for embodied AI models and WAMs.<n>The runtime provides multi-rate execution, latency-first fused inference, and operator and I/O support.<n>We show that Embodied$.$ improves deployment efficiency while preserving high accuracy across diverse embodied model architectures.
  arXiv  Detail & Related papers  (2026-07-02T17:58:28Z)
• MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving [54.57163800903507]
  We present MindVLA-U1, the first unified streaming VLA architecture for autonomous driving.<n>A unified VLM backbone produces AR language tokens and flow-matching continuous action trajectories in a single forward pass over one shared representation.<n>On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time.
  arXiv  Detail & Related papers  (2026-05-12T18:09:42Z)
• VLA Foundry: A Unified Framework for Training Vision-Language-Action Models [11.774960393195052]
  We present VLA Foundry, an open-source framework that unifies LLM, VLM, and VLA training in a single stack.<n>VLA Foundry supports both from-scratch training and pretrained backbones from Hugging Face.<n>We evaluate closed-loop policy performance of both models on LBM Eval, an open-data, open-source simulator.
  arXiv  Detail & Related papers  (2026-04-21T17:51:51Z)
• StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing [0.0]
  Building generalist embodied agents requires integrating perception, language understanding, and action.<n>We present StarVLA, an open-source for Vision-Language-Action research.
  arXiv  Detail & Related papers  (2026-04-06T17:59:21Z)
• vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models [58.633451339058986]
  VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository.<n>We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution.<n>A complete evaluation requires only two commands: vla eval serve and vla eval run.
  arXiv  Detail & Related papers  (2026-03-14T14:38:53Z)
• LiteVLA-Edge: Quantized On-Device Multimodal Control for Embedded Robotics [0.6119773373677944]
  We present LiteVLA-Edge, a deployment-oriented VLA pipeline for fully on-device inference on Jetson Orin-class hardware.<n>Our approach combines supervised image-to-action fine-tuning in FP32 with post-training 4-bit GGUF quantization and GPU-accelerated inference.<n>Under our configuration, LiteVLA-Edge achieves a mean end-to-end runtime of 150.5,ms (approximately 6.6,Hz) while operating entirely offline.
  arXiv  Detail & Related papers  (2026-03-03T03:20:52Z)
• VibeTensor: System Software for Deep Learning, Fully Generated by AI Agents [42.56489784841984]
  "fully generated" refers to code provenance: implementation changes were produced and applied as agent-proposed diffs.<n>We describe the architecture, summarize the workflow used to produce and validate the system, and evaluate the artifact.
  arXiv  Detail & Related papers  (2026-01-21T19:29:00Z)
• VLAgents: A Policy Server for Efficient VLA Inference [10.731074886477515]
  We introduce VLAgents, a modular policy server that abstracts VLA inferencing behind a unified Gymnasium-style protocol.<n>Its communication layer transparently adapts to the context by supporting both zero-copy shared memory for high-speed simulation and compressed streaming for remote hardware.<n>In a benchmark with both local and remote communication, we demonstrate how it outperforms the default policy servers provided by OpenVLA, OpenPi, and LeRobot.
  arXiv  Detail & Related papers  (2026-01-16T12:58:59Z)
• OpenVLA: An Open-Source Vision-Language-Action Model [131.74098076670103]
  We introduce OpenVLA, an open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA shows strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate. We release model checkpoints, fine-tuning notebooks, and our PyTorch with built-in support for training VLAs at scale on Open X-Embodiment datasets.
  arXiv  Detail & Related papers  (2024-06-13T15:46:55Z)
• MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices [73.46317110474064]
  MobileVLM is a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It comprises a set of language models at the scale of 1.4B and 2.7B parameters, trained from scratch, a multimodal vision model that is pre-trained in the CLIP fashion.
  arXiv  Detail & Related papers  (2023-12-28T08:21:24Z)
• Tracking Anything with Decoupled Video Segmentation [87.07258378407289]
  We develop a decoupled video segmentation approach (DEVA) It is composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks.
  arXiv  Detail & Related papers  (2023-09-07T17:59:41Z)
• TarViS: A Unified Approach for Target-based Video Segmentation [115.5770357189209]
  TarViS is a novel, unified network architecture that can be applied to any task that requires segmenting a set of arbitrarily defined 'targets' in video. Our approach is flexible with respect to how tasks define these targets, since it models the latter as abstract 'queries' which are then used to predict pixel-precise target masks. To demonstrate its effectiveness, we apply TarViS to four different tasks, namely Video Instance (VIS), Video Panoptic (VPS), Video Object (VOS) and Point Exemplar-guided Tracking (PET)
  arXiv  Detail & Related papers  (2023-01-06T18:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.