Patch Policy: Efficient Embodied Control via Dense Visual Representations

• URL: http://arxiv.org/abs/2607.18236v1
• Date: Mon, 20 Jul 2026 17:59:41 GMT
• Title: Patch Policy: Efficient Embodied Control via Dense Visual Representations
• Abstract summary: We develop Patch Policy to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM.<n>Our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations.<n>It surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters.
• Score: 52.199962246592285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io

Related papers

• DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation [54.860960182207236]
  We introduce DynaFLIP, a multimodal pre-training framework that pushes motion understanding upstream into perception.<n>We construct image-language-3D flow triplets from heterogeneous human and robot videos, and use these triplets as training-time supervision to shape an image-only encoder.<n>Our results suggest that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.
  arXiv  Detail & Related papers  (2026-05-28T17:59:53Z)
• Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models [19.87835200940665]
  TPRL learns adaptive pruning trajectories through language-guided sequential optimization tied directly to end-task performance.<n>We formulate visual token pruning as a sequential decision process with explicit state transitions.<n>We employ a self-supervised autoencoder to compress visual tokens into a compact state representation for efficient policy learning.
  arXiv  Detail & Related papers  (2026-03-11T09:52:44Z)
• IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance [35.90972175891154]
  Many Vision-Language-Action (VLA) models flatten image patches into a 1D token sequence, weakening the 2D spatial cues needed for precise manipulation.<n>We introduce a lightweight, training-free method that improves spatial understanding by exploiting affinity hints already available in the model's built-in vision.
  arXiv  Detail & Related papers  (2026-01-22T18:57:13Z)
• See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection [51.59559387222532]
  Recent advances in end-to-end autonomous driving show that policies trained on patch-aligned features generalize better to Out-of-Distribution (OOD)<n>We present $2.4-Patch-Selection (SPS), a simple yet effective approach for learning policies that are more robust, generalizable, and efficient.
  arXiv  Detail & Related papers  (2026-01-15T18:58:33Z)
• When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models [81.7618160628979]
  Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored.<n>We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space.<n> Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints.
  arXiv  Detail & Related papers  (2025-11-26T09:16:32Z)
• MemER: Scaling Up Memory for Robot Control via Experience Retrieval [46.5398413633767]
  Humans routinely rely on memory to tasks, yet most robot policies lack this capability.<n>We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevants from its experience.<n>Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory.
  arXiv  Detail & Related papers  (2025-10-23T08:26:17Z)
• Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs [88.68484904214142]
  We introduce Patch-as-Decodable Token (PaDT) to generate both textual and diverse visual outputs.<n>Central to PaDT are Visual Reference Tokens (VRTs), derived from visual patch embeddings of query images.<n>We show PaDT consistently achieves state-of-the-art performance, even compared with significantly larger MLLM models.
  arXiv  Detail & Related papers  (2025-10-02T12:23:57Z)
• PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies [32.6823407205603]
  We present PEEK (Policy-agnostic Extraction of Essential Keypoints), which fine-tunes vision-language models (VLMs) to predict a unified point-based intermediate representation.<n>These annotations are directly overlaid onto robot observations, making the representation policy-agnostic and transferable across architectures.<n>In real-world evaluations, PEEK consistently boosts zero-shot generalization, including a 41.4x real-world improvement for a 3D policy trained only in simulation.
  arXiv  Detail & Related papers  (2025-09-22T18:10:14Z)
• FLARE: Robot Learning with Implicit World Modeling [87.81846091038676]
  $textbfFLARE$ integrates predictive latent world modeling into robot policy learning.<n>$textbfFLARE$ achieves state-of-the-art performance, outperforming prior policy learning baselines by up to 26%.<n>Our results establish $textbfFLARE$ as a general and scalable approach for combining implicit world modeling with high-frequency robotic control.
  arXiv  Detail & Related papers  (2025-05-21T15:33:27Z)
• Zero-Shot Visual Generalization in Robot Manipulation [0.13280779791485384]
  Current approaches often sidestep the problem by relying on invariant representations such as point clouds and depth.<n>Disentangled representation learning has recently shown promise in enabling vision-based reinforcement learning policies to be robust to visual distribution shifts.<n>We demonstrate zero-shot adaptability to visual perturbations in both simulation and on real hardware.
  arXiv  Detail & Related papers  (2025-05-16T22:01:46Z)
• Any-point Trajectory Modeling for Policy Learning [64.23861308947852]
  We introduce Any-point Trajectory Modeling (ATM) to predict future trajectories of arbitrary points within a video frame. ATM outperforms strong video pre-training baselines by 80% on average. We show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology.
  arXiv  Detail & Related papers  (2023-12-28T23:34:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.