RepSAM: Bridging Foundation Models to Robotic Vision via Representation-Guided Adaptation

• URL: http://arxiv.org/abs/2605.25495v1
• Date: Mon, 25 May 2026 06:56:42 GMT
• Title: RepSAM: Bridging Foundation Models to Robotic Vision via Representation-Guided Adaptation
• Abstract summary: RepSAM is a representation-guided parameter-efficient fine-tuning framework for adapting foundation models to robotic vision.<n>RepSAM achieves 97.9% of full fine-tuning performance (89.0% vs. 90.9% mIoU) while reducing trainable parameters by 158x (from 632M to 4.0M).
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robotic perception in unstructured environments remains challenging despite the zero-shot capabilities of foundation models such as SAM. This work attributes performance degradation to non-uniform representation shifts across transformer layers: shallow layers exhibit substantial domain gaps (CKA < 0.5), whereas deep layers transfer effectively (CKA > 0.7). Based on this observation, we propose RepSAM, a representation-guided parameter-efficient fine-tuning (PEFT) framework for adapting foundation models to robotic vision. RepSAM employs a theoretically grounded CKA-guided rank allocation strategy combined with a multi-modal fusion module for robust handling of challenging robotic scenarios, including transparent objects and cluttered scenes. Experimental evaluation across six benchmarks and robotic manipulation tasks demonstrates that RepSAM achieves 97.9% of full fine-tuning performance (89.0% vs. 90.9% mIoU) while reducing trainable parameters by 158x (from 632M to 4.0M). RepSAM outperforms DoRA by 7.9% mIoU with just 4 hours of training on a single A100 GPU (a 96x reduction from full fine-tuning, which takes 384 GPU-hours). These improvements are statistically significant (p < 0.01) and translate to a 12.0% absolute improvement in robotic manipulation success rates over the LoRA (RGB) baseline.

Related papers

• Pelican-Sim 1.0: A General World Model Simulator for Embodied Intelligence [22.0636262013758]
  Pelican-Sim 1.0 is a general world model simulator for embodied intelligence.<n>It predicts future observations from visual context and robot actions to support downstream learning and decision making.
  arXiv  Detail & Related papers  (2026-09-10T17:51:01Z)
• LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation [53.88286427894783]
  Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context.<n>Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots.<n>We present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B- parameter local VLM for edge execution on embedded GPU devices.
  arXiv  Detail & Related papers  (2026-06-26T09:11:59Z)
• CAX-Agent: A Lightweight Agent Harness for Reliable APDL Automation [7.355373109826612]
  This paper presents the architecture of CAX-Agent, a lightweight agent harness for MAPDL automation.<n>We evaluate three recovery strategies (no_recovery, rule_only, and model_only) on 50 standard structural benchmarks.<n>Model_only achieves the best completion rate (0.9267), task score (3.59/4), total score (9.16/10), and zero-intervention rate (0.84), outperforming rule_only (0.7733, 3.17/4, 7.03/10, 0.00) and no_recovery (0.6933, 2.74/4, 5.60/10, 0.00) with large effect sizes
  arXiv  Detail & Related papers  (2026-05-12T14:46:34Z)
• MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learning [4.076622576916001]
  Multi-modal Transformer-based 3D Affordances and Reinforcement Learning (RL)<n>This paper proposes MTA-RL, the first framework that bridges perception and control through Multi-modal Transformer-based 3D Affordances and Reinforcement Learning (RL)
  arXiv  Detail & Related papers  (2026-05-11T08:28:55Z)
• From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation [32.41847293364159]
  PRIMO R1 is a 7B framework that transforms video MLLMs into active "Critics"<n>We leverage outcome-based Reinforcement Learning to incentivize explicit Chain-of-Thought generation for progress estimation.<n>We establish state-of-the-art performance on RoboFail benchmark with 67.0% accuracy, surpassing closed-source models like OpenAI o1 by 6.2%.
  arXiv  Detail & Related papers  (2026-03-16T17:53:28Z)
• Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer [59.02729900344616]
  GPU-accelerated, photorealistic simulation has opened a scalable data-generation path for robot learning.<n>We develop a teacher-student-bootstrap learning framework for vision-based humanoid loco-manipulation.<n>This represents the first humanoid sim-to-real policy capable of diverse articulated loco-manipulation using pure RGB perception.
  arXiv  Detail & Related papers  (2025-11-30T20:07:13Z)
• Contrastive Representation Regularization for Vision-Language-Action Models [64.10170453130324]
  We introduce Robot State-aware Contrastive Loss (RS-CL), a representation regularization for Vision-Language-Action (VLA) models.<n>In particular, RS-CL aligns the representations more closely with the robot's proprioceptive states, by using relative distances between the states as soft supervision.<n>Our empirical results demonstrate that RS-CL substantially improves the manipulation performance of state-of-the-art VLA models.
  arXiv  Detail & Related papers  (2025-10-02T06:41:22Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• Diversity-Guided MLP Reduction for Efficient Large Vision Transformers [62.33249256133204]
  Transformer models achieve excellent scaling property, where the performance is improved with the increment of model capacity.<n>Large-scale model parameters lead to an unaffordable cost of computing and memory.<n>We propose a Diversity-Guided Reduction (DGMR) method to significantly reduce the parameters of large vision transformers.
  arXiv  Detail & Related papers  (2025-06-10T08:59:27Z)
• From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation [35.79160868966466]
  We propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning.<n>Our approach combines a hierarchical data pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals.<n>We show that FSD achieves 40.6% success rate in SimplerEnv and 72% success rate across 8 real-world tasks, outperforming the strongest baseline by 30%.
  arXiv  Detail & Related papers  (2025-05-13T13:20:46Z)
• Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web [69.6913064185993]
  Language model agents (LMA) emerged as a promising paradigm on muti-step decision making tasks.<n>Despite the promise, their performance on real-world applications is still underexplored.<n>We show that while existing LMAs achieve 94.0% average success rate on base tasks, their performance degrades to 24.9% success rate on compositional tasks.
  arXiv  Detail & Related papers  (2023-11-30T17:50:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.