Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

• URL: http://arxiv.org/abs/2607.08448v3
• Date: Wed, 15 Jul 2026 16:40:12 GMT
• Title: Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
• Abstract summary: We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive.<n>The framework learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models.<n>Across tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points.
• Score: 23.94713093511847
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R.

Related papers

• HAF: Adapting Generalist VLAs to Humanoid Whole-Body Loco-manipulation via Hierarchical Action Flow and Spectral Latent RL [56.014913786737104]
  We introduce HAF (Humanoid Adaptation Framework) to transfer off-the-shelf generalist VLA foundation models to humanoid whole-body loco-manipulation.<n> HAF-VLA is a hierarchical action-flow generator built on a pretrained flow-matching VLA.<n>HaF-Steer is a latent offline-to-online RL pipeline that leverages flow-matching invertibility and DCT-based dimensionality reduction.
  arXiv  Detail & Related papers  (2026-08-17T17:22:33Z)
• ViTaR: Visuo-Tactile Residual Adaptation for Foundation VLA Manipulation [4.374415463111511]
  We introduce ViTaR, which reframes tactile feedback from an action-generating perceptual input to an execution modulator.<n>On the UniVTAC benchmark spanning seven contact-rich tasks, ViTaR achieves 61.3% average success, a 30.6 percentage-point improvement over its frozen VLA base.
  arXiv  Detail & Related papers  (2026-08-16T15:46:11Z)
• TORL-VLA: Tactile Guided Online Reinforcement Learning for Contact-Rich Manipulation [24.661831588991898]
  TORL-VLA is an online reinforcement learning framework that couples tactile feedback with policy refinement for contact-rich manipulation.<n>Our method introduces a tactile-derived wrench-aware VLA to predict reference actions and future wrench sequences.<n>To stabilize learning from mixed exploratory policy-generated and human-intervention data, we introduce an intervention-censored critic.
  arXiv  Detail & Related papers  (2026-06-08T11:05:05Z)
• PHASER: Phase-Aware and Semantic Experience Replay for Vision-Language-Action Models [34.954808072081676]
  We introduce PHASER, an architecture-agnostic continual learning framework.<n>PhaseR employs a phase-centric capacity allocation to guarantee equal memory support for all sub-skills.<n>We also integrate Auto-PC, a lightweight pipeline combining unsupervised action-signal change-point detection with VLM-based semantic verification.
  arXiv  Detail & Related papers  (2026-06-02T13:04:15Z)
• PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations [66.94988600664574]
  We present textbfPRTS (textbfPrimitive textbfReasoning and textbfTasking textbfSystem), a VLA foundation model that reformulates pretraining through Goal-Conditioned Reinforcement Learning.
  arXiv  Detail & Related papers  (2026-04-30T06:14:02Z)
• AnchorRefine: Synergy-Manipulation Based on Trajectory Anchor and Residual Refinement for Vision-Language-Action Models [60.04879435087352]
  Vision-language-action (VLA) policies generate actions within a single unified space.<n>We propose AnchorRefine, a hierarchical framework that factorizes VLA action modeling into trajectory anchor and residual refinement.<n> Experiments on LIBERO, CALVIN, and real-robot tasks demonstrate that AnchorRefine consistently improves both regression-based and diffusion-based VLA backbones.
  arXiv  Detail & Related papers  (2026-04-20T04:25:24Z)
• Jump-Start Reinforcement Learning with Vision-Language-Action Regularization [1.2599533416395767]
  Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation.<n>Current limitations hinder their direct use in fast and precise manipulation.<n>We propose Vision-Language-Action Jump-Starting (VLAJS) to improve exploration and learning efficiency.
  arXiv  Detail & Related papers  (2026-04-15T11:17:54Z)
• Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA [62.16042475700567]
  We introduce IMCopilot, a shared-autonomy assistant to simplify teleoperation data collection.<n>We present MoDE-VLA, an architecture that seamlessly integrates heterogeneous force and tactile modalities into a pretrained VLA backbone.<n>We validate our approach on four tasks of escalating complexity, demonstrating doubled success rate improvement over the baseline in dexterous contact-rich tasks.
  arXiv  Detail & Related papers  (2026-03-09T09:02:30Z)
• LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies [54.150202739999806]
  LiLo-VLA is a modular framework capable of zero-shot modularity to novel long-horizon tasks without ever being trained on them.<n>We introduce a 21-task simulation benchmark consisting of two challenging suites: LIBERO-Long++ and Ultra-Long.<n>In these simulations, LiLo-VLA achieves a 69% average success rate, outperforming Pi0.5 by 41% and OpenVLA-OFT by 67%.
  arXiv  Detail & Related papers  (2026-02-25T03:33:39Z)
• From Knowing to Doing Precisely: A General Self-Correction and Termination Framework for VLA models [42.553888395227766]
  We propose a lightweight, training-free framework for vision-language-action (VLA) models.<n>VLA-SCT operates as a self-correcting control loop, combining data-driven action refinement with conditional logic for termination.<n>Our method achieves consistent improvements across all datasets in the LIBERO benchmark.
  arXiv  Detail & Related papers  (2026-02-02T08:44:40Z)
• Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulation [25.496791183299692]
  Affordance Field Intervention (AFI) is a lightweight hybrid framework that uses SAFs as an on-demand plug-in to guide VLA behavior.<n>Our method achieves an average improvement of 23.5% across different VLA backbones.
  arXiv  Detail & Related papers  (2025-12-08T11:57:13Z)
• Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policies [62.653984010274485]
  Vision-Language-Action (VLA) models adapt large vision-language backbones to map images and instructions into robot actions.<n> prevailingAs either generate actions auto-regressively in a fixed left-to-right order or attach separate or diffusion heads outside the backbone.<n>We present Discrete Diffusion VLA, a unified-transformer policy that models discretized action chunks with discrete diffusion.
  arXiv  Detail & Related papers  (2025-08-27T17:39:11Z)
• Adversarial Attacks on Robotic Vision Language Action Models [118.02118618146568]
  We study adversarial attacks on vision-language-action models (VLAs)<n>Our main algorithmic contribution is the adaptation and application of LLM jailbreaking attacks to obtain complete control authority.<n>This differs significantly from LLM jailbreaking literature, as attacks in the real world do not have to be semantically linked to notions of harm.
  arXiv  Detail & Related papers  (2025-06-03T19:43:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.