DINOv3-Diffusion Policy: Self-Supervised Large Visual Model for Visuomotor Diffusion Policy Learning

• URL: http://arxiv.org/abs/2509.17684v1
• Date: Mon, 22 Sep 2025 12:27:26 GMT
• Title: DINOv3-Diffusion Policy: Self-Supervised Large Visual Model for Visuomotor Diffusion Policy Learning
• Abstract summary: We investigate whether a purely self-supervised encoder can match or surpass conventional supervised ImageNet-pretrained backbones.<n>We find that finetuned DINOv3 matches or exceeds ResNet-18 on several tasks.<n>Compared to using ResNet18 as a backbone, our approach with DINOv3 achieves up to a 10% absolute increase in test-time success rates.
• Score: 4.9400495816263765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper evaluates DINOv3, a recent large-scale self-supervised vision backbone, for visuomotor diffusion policy learning in robotic manipulation. We investigate whether a purely self-supervised encoder can match or surpass conventional supervised ImageNet-pretrained backbones (e.g., ResNet-18) under three regimes: training from scratch, frozen, and finetuned. Across four benchmark tasks (Push-T, Lift, Can, Square) using a unified FiLM-conditioned diffusion policy, we find that (i) finetuned DINOv3 matches or exceeds ResNet-18 on several tasks, (ii) frozen DINOv3 remains competitive, indicating strong transferable priors, and (iii) self-supervised features improve sample efficiency and robustness. These results support self-supervised large visual models as effective, generalizable perceptual front-ends for action diffusion policies, motivating further exploration of scalable label-free pretraining in robotic manipulation. Compared to using ResNet18 as a backbone, our approach with DINOv3 achieves up to a 10% absolute increase in test-time success rates on challenging tasks such as Can, and on-the-par performance in tasks like Lift, PushT, and Square.

Related papers

• Robot Trajectron V3: A Probabilistic Shared Control Framework for SE(3) Manipulation [5.090943130063491]
  RT-V3 is a shared control framework designed for $SE(3)$ grasping tasks.<n>It learns a prior over user intent and combines it with real-time user commands to estimate the posterior intent distribution.<n>Experiments demonstrate that RT-V3 achieves high accuracy in trajectory prediction and competitive performance in reactive planning.
  arXiv  Detail & Related papers  (2026-07-10T11:52:18Z)
• Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulation [67.16145181790522]
  This paper tackles spatial perception and manipulation challenges in Vision-Language-Action (VLA) models.<n>We propose a Geometry-Guided Gated Transformer (G3T) that aligns multi-view features under 3D geometric guidance.<n>To improve action learning efficiency, we introduce Action Manifold Learning (AML), which directly predicts actions on the valid action manifold.
  arXiv  Detail & Related papers  (2026-05-12T09:21:29Z)
• Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation [53.750389076941396]
  Diffusion-based visuomotor policies capture multimodal action distributions through iterative denoising, but their high inference latency limits real-time robotic control.<n>We propose Ada3Drift, which learns a training-time drifting field that attracts predicted actions toward expert demonstration modes.<n>Ada3Drift achieves state-of-the-art performance while requiring $10times$ fewer function evaluations than diffusion-based alternatives.
  arXiv  Detail & Related papers  (2026-03-12T14:36:53Z)
• X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learning [31.894643265305476]
  Visuomotor policies often leverage large pre-trained Vision Transformers (ViTs) for their powerful generalization capabilities.<n>We introduce X-Distill, a simple yet highly effective method that synergizes the strengths of both architectures.<n>Our approach involves an offline, cross-architecture knowledge distillation, transferring the rich visual representations of a large, frozen DINOv2 teacher to a compact ResNet-18 student.
  arXiv  Detail & Related papers  (2026-01-16T13:15:55Z)
• VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models [11.02910353976723]
  We present VLAD-Grasp, a vision-language model assisted zero-shot approach for detecting grasps.<n>Unlike prior work, our approach is training-free and does not rely on curated grasp datasets.<n>We further demonstrate zero-shot generalization to novel real-world objects on a Franka Research 3 robot.
  arXiv  Detail & Related papers  (2025-11-08T01:47:40Z)
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency [255.9417257812203]
  InternVL 3.5 is a new family of open-source multimodal models that significantly advances versatility, reasoning capability, and inference efficiency.<n>Key innovation is the Cascade Reinforcement Learning framework, which enhances reasoning through a two-stage process.<n>Our largest model, i.e., InternVL3.5-241B-A28B, attains state-of-the-art results among open-source MLLMs across general multimodal, reasoning, text, and agentic tasks.
  arXiv  Detail & Related papers  (2025-08-25T17:58:17Z)
• Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Reward [85.84943447589511]
  This paper introduces AC3 (Actor-Critic for Continuous Chunks), a novel RL framework that learns to generate high-dimensional, continuous action sequences.<n>To make this learning process stable and data-efficient, AC3 incorporates targeted stabilization mechanisms for both the actor and the critic.
  arXiv  Detail & Related papers  (2025-08-15T01:27:15Z)
• One RL to See Them All: Visual Triple Unified Reinforcement Learning [92.90120580989839]
  We propose V-Triune, a Visual Triple Unified Reinforcement Learning system that enables visual reasoning and perception tasks within a single training pipeline.<n>V-Triune comprises triple complementary components: Sample-Level Datashelf (to unify diverse task inputs), Verifier-Level Reward (to deliver custom rewards via specialized verifiers).<n>We introduce a novel Dynamic IoU reward, which provides adaptive, progressive, and definite feedback for perception tasks handled by V-Triune.
  arXiv  Detail & Related papers  (2025-05-23T17:41:14Z)
• MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning [17.437573206368494]
  Visual deep reinforcement learning (RL) enables robots to acquire skills from visual input for unstructured tasks.<n>We present MENTOR, a method that improves both the architecture and optimization of RL agents.<n>MenTOR outperforms state-of-the-art methods across three simulation benchmarks and achieves an average of 83% success rate on three challenging real-world robotic manipulation tasks.
  arXiv  Detail & Related papers  (2024-10-19T04:31:54Z)
• Foundation Model-Powered 3D Few-Shot Class Incremental Learning via Training-free Adaptor [9.54964908165465]
  This paper introduces a new method to tackle the Few-Shot Continual Incremental Learning problem in 3D point cloud environments. We leverage a foundational 3D model trained extensively on point cloud data. Our approach uses a dual cache system: first, it uses previous test samples based on how confident the model was in its predictions to prevent forgetting, and second, it includes a small number of new task samples to prevent overfitting.
  arXiv  Detail & Related papers  (2024-10-11T20:23:00Z)
• ACTRESS: Active Retraining for Semi-supervised Visual Grounding [52.08834188447851]
  A previous study, RefTeacher, makes the first attempt to tackle this task by adopting the teacher-student framework to provide pseudo confidence supervision and attention-based supervision. This approach is incompatible with current state-of-the-art visual grounding models, which follow the Transformer-based pipeline. Our paper proposes the ACTive REtraining approach for Semi-Supervised Visual Grounding, abbreviated as ACTRESS.
  arXiv  Detail & Related papers  (2024-07-03T16:33:31Z)
• Stitched ViTs are Flexible Vision Backbones [51.441023711924835]
  We are inspired by stitchable neural networks (SN-Net) to produce a single model that covers richworks by stitching pretrained model families. We introduce SN-Netv2, a systematically improved model stitching framework to facilitate downstream task adaptation. SN-Netv2 demonstrates superior performance over SN-Netv1 on downstream dense predictions and shows strong ability as a flexible vision backbone.
  arXiv  Detail & Related papers  (2023-06-30T22:05:34Z)
• Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation [25.47207030637466]
  Large models pretrained on common visual learning tasks can provide useful representations for a wide range of specialized perception problems. We introduce "lossless adaptation" to address this shortcoming of classical fine-tuning. We demonstrate that appropriate placement of our parameter efficient adapters can significantly reduce the performance gap between frozen pretrained representations and full end-to-end fine-tuning.
  arXiv  Detail & Related papers  (2023-04-13T15:06:28Z)
• Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives [89.34229413345541]
  We propose a conditioning scheme which avoids pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion. We report significant improvements in task success over representative MPC and IL baselines.
  arXiv  Detail & Related papers  (2020-03-19T15:04:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.