PVI: Plug-in Visual Injection for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.12772v1
• Date: Fri, 13 Mar 2026 08:16:53 GMT
• Title: PVI: Plug-in Visual Injection for Vision-Language-Action Models
• Abstract summary: VLAs that pair a pretrained VLM with a flow-matching action expert have emerged as a strong paradigm for language-conditioned manipulation.<n>We propose Plug-in Visual Injection (PVI), a lightweight encoder-agnostic module that attaches to a pretrained action expert and injects auxiliary visual representations via zero-d residual pathways.
• Score: 8.641415997612066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: VLA architectures that pair a pretrained VLM with a flow-matching action expert have emerged as a strong paradigm for language-conditioned manipulation. Yet the VLM, optimized for semantic abstraction and typically conditioned on static visual observations, tends to attenuate fine-grained geometric cues and often lacks explicit temporal evidence for the action expert. Prior work mitigates this by injecting auxiliary visual features, but existing approaches either focus on static spatial representations or require substantial architectural modifications to accommodate temporal inputs, leaving temporal information underexplored. We propose Plug-in Visual Injection (PVI), a lightweight, encoder-agnostic module that attaches to a pretrained action expert and injects auxiliary visual representations via zero-initialized residual pathways, preserving pretrained behavior with only single-stage fine-tuning. Using PVI, we obtain consistent gains over the base policy and a range of competitive alternative injection strategies, and our controlled study shows that temporal video features (V-JEPA2) outperform strong static image features (DINOv2), with the largest gains on multi-phase tasks requiring state tracking and coordination. Real-robot experiments on long-horizon bimanual cloth folding further demonstrate the practicality of PVI beyond simulation.

Related papers

• Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models [24.27863433568267]
  We propose Robust-WAM, a general post-training method for video-generation-based WAMs.<n>We employ learnable query tokens to bring future-scene semantics into the action stream by aligning their output hidden states with the semantic foresight of future ground-truth frames.<n> Experiments on out-of-distribution generalization simulation benchmarks and a real-robot setup show that our Robust-WAM consistently improves the success rates of multiple WAM baselines without sacrificing in-distribution performance.
  arXiv  Detail & Related papers  (2026-08-06T11:31:56Z)
• DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Images [50.96240625673857]
  DroneFINE is a novel PEFT paradigm comprising two domain-aware complementary modules tailored for VLM-based drone image detectors.<n>DroneFINE significantly outperforms existing PEFT methods and achieves performance comparable to full fine-tuning.
  arXiv  Detail & Related papers  (2026-07-01T02:27:40Z)
• Action with Visual Primitives [36.230909065494345]
  Vision-Language-Action (VLA) models have emerged as a promising paradigm for generalist robotic manipulation.<n>We introduce AVP (Action with Visual Primitives), an end-to-end architecture that implements this visual-primitive-centric interface.<n>We show that AVP improves the success rate by 27.61% over pi_0.5 and outperforms other recent methods.
  arXiv  Detail & Related papers  (2026-05-21T08:52:47Z)
• Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs [66.15429821530503]
  Persistent Visual Memory is a lightweight learnable module designed to strengthen sustained, on-demand access to visual evidence.<n>Experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead.<n>In-depth analysis reveals that PVM shows improved robustness in longer generations and accelerates internal prediction convergence.
  arXiv  Detail & Related papers  (2026-05-01T17:54:37Z)
• Visual prompting reimagined: The power of the Activation Prompts [72.85146015928626]
  We introduce a concept, termed activation prompt (AP), which extends the scope of the input-level VP.<n>AP is closely related to normalization tuning in convolutional neural networks and vision transformers.<n>We show that AP is closely related to normalization tuning in convolutional neural networks and vision transformers.
  arXiv  Detail & Related papers  (2026-04-07T20:28:24Z)
• BEVPredFormer: Spatio-temporal Attention for BEV Instance Prediction in Autonomous Driving [1.5327485092552822]
  We introduce BEVPredFormer, a novel camera-only architecture for BEV prediction.<n>It uses attention-based temporal processing to improve temporal and spatial comprehension of the scene.<n>It was evaluated on the nuScenes dataset and was on par or surpassed State-Of-The-Art methods.
  arXiv  Detail & Related papers  (2026-04-03T09:58:42Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• ESPADA: Execution Speedup via Semantics Aware Demonstration Data Downsampling for Imitation Learning [18.435889278351297]
  ESPADA is a semantically aware framework that segments demonstrations using a VLM-LLM pipeline with 3D gripper-object relations.<n>To scale from a single annotated episode to the full dataset, ESPADA propagates segment labels via Dynamic Time Warping.<n> ESPADA achieves approximately a 2x speed-up while maintaining success rates, narrowing the gap between human demonstrations and efficient robot control.
  arXiv  Detail & Related papers  (2025-12-08T10:08:33Z)
• Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models [13.32858759983739]
  Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating text inconsistent with visual inputs.<n>Existing inference-time interventions to mitigate this issue present a challenging trade-off.<n>We present Residual-Update Directed DEcoding Regulation (RUDDER), a framework that steers LVLMs towards visually-grounded generation.
  arXiv  Detail & Related papers  (2025-11-13T13:29:38Z)
• Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos [58.156141601478794]
  Multi-object tracking (UAVT) aims to track multiple objects while maintaining consistent identities across frames of a given video.<n>Existing methods typically model motion cues and appearance separately, overlooking their interplay and resulting in suboptimal tracking performance.<n>We propose AMOT, which exploits appearance and motion cues through two key components: an Appearance-Motion Consistency (AMC) matrix and a Motion-aware Track Continuation (MTC) module.
  arXiv  Detail & Related papers  (2025-08-03T12:06:47Z)
• VITA: Vision-to-Action Flow Matching Policy [27.783132918359545]
  VITA is a noise-free and conditioning-free policy learning framework.<n>It maps visual representations to latent actions using flow matching.<n>We evaluate VITA on 8 simulation and 2 real-world tasks from ALOHA and Robomimic.
  arXiv  Detail & Related papers  (2025-07-17T15:41:57Z)
• From Sight to Insight: Unleashing Eye-Tracking in Weakly Supervised Video Salient Object Detection [60.11169426478452]
  This paper aims to introduce fixation information to assist the detection of salient objects under weak supervision.<n>We propose a Position and Semantic Embedding (PSE) module to provide location and semantic guidance during the feature learning process.<n>An Intra-Inter Mixed Contrastive (MCII) model improves thetemporal modeling capabilities under weak supervision.
  arXiv  Detail & Related papers  (2025-06-30T05:01:40Z)
• AssistPDA: An Online Video Surveillance Assistant for Video Anomaly Prediction, Detection, and Analysis [52.261173507177396]
  We introduce AssistPDA, the first online video anomaly surveillance assistant (VAPDA) that unifies anomaly prediction, detection, and analysis (VAPDA) within a single framework.<n> AssistPDA enables real-time inference on streaming videos while supporting interactive user engagement.<n>We also introduce a novel event-level anomaly prediction task, enabling proactive anomaly forecasting before anomalies fully unfold.
  arXiv  Detail & Related papers  (2025-03-27T18:30:47Z)
• LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry [53.5449912019877]
  We present the Long-term Effective Any Point Tracking (LEAP) module.<n>LEAP innovatively combines visual, inter-track, and temporal cues with mindfully selected anchors for dynamic track estimation.<n>Based on these traits, we develop LEAP-VO, a robust visual odometry system adept at handling occlusions and dynamic scenes.
  arXiv  Detail & Related papers  (2024-01-03T18:57:27Z)
• Off-policy Imitation Learning from Visual Inputs [83.22342811160114]
  We propose OPIfVI, which is composed of an off-policy learning manner, data augmentation, and encoder techniques. We show that OPIfVI is able to achieve expert-level performance and outperform existing baselines.
  arXiv  Detail & Related papers  (2021-11-08T09:06:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.