Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots

• URL: http://arxiv.org/abs/2607.13522v1
• Date: Wed, 15 Jul 2026 07:21:00 GMT
• Title: Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots
• Abstract summary: Force and contact leave almost no trace in a single frame, and raw vision features read force at $R2$ at or below $0.10$ on every robot we test.<n>We present Kepler-Encoder-v0.1, a robot-first multimodal encoder that treats robot state as a modality and fuses vision, proprioception, and force/torque into a single shared latent.<n>On held-out scenes, the vision-only latent recovers end-effector state, and force in particular, significantly above both raw frozen-ViT features and a compute-matched vision-only control on every sensored robot.
• Score: 7.140499031711496
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A robot must understand the state of its own body, but a camera sees only part of it. Force and contact leave almost no trace in a single frame, and raw vision features read force at $R^2$ at or below $0.10$ on every robot we test. We present Kepler-Encoder-v0.1, a robot-first multimodal encoder that treats robot state as a modality and fuses vision, proprioception, and force/torque into a single shared latent with a learned-query cross-attention layer, trained self-supervised by masked cross-modal prediction under the LeJEPA/SIGReg objective. At evaluation only vision enters, which poses a sharp question. Does fusing state into training make the vision-only latent carry anything the pixels do not already contain? On the RH20T corpus the answer is yes, precisely where the camera is weakest. On held-out scenes, the vision-only latent recovers end-effector state, and force in particular, significantly above both raw frozen-ViT features and a compute-matched vision-only control on every sensored robot, though absolute force recovery at a single timestep is modest; on motor state, which the camera largely sees, it is statistically tied with the strongest vision baselines, and it is the only feature whose latent geometry tracks state. A single embodiment-agnostic encoder covers four robots, and a data-matched control shows this breadth reflects embodiment diversity rather than data volume. The frozen latent is directly useful. Its own cross-modal prediction error is a training-free invalid-state monitor (AUROC $0.90$ on out-of-range states, $0.69$ on scene-swapped states), and a diffusion decoder (PixNerd) reconstructs the camera frame from the latent, confirming the spatial compression preserves world-state. This report validates the single-timestep case; native-rate temporal fusion is the next step.

Related papers

• DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery [51.92897182932217]
  Egocentric video offers scalable manipulation data for embodied AI.<n>However, recovering metric 3D hand trajectories is challenging due to severe object occlusion and frequent out-of-sight gaps.<n>We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent and decodes them with a Bidirectional Spatiotemporal Decoder.
  arXiv  Detail & Related papers  (2026-08-20T17:46:24Z)
• Robostral Navigate [138.74472687550374]
  Robostral Navigate is a vision-language model for navigation systems at scale.<n>It consumes only a stream of monocular RGB images and predicts waypoints by pointing to the next target location in the current camera view.<n>We generate 2.4 million trajectories across 350k simulated scenes to reduce the reliance on real-world data collection.
  arXiv  Detail & Related papers  (2026-07-22T23:13:05Z)
• DemoBridge: A Simulation-in-the-Loop Toolkit for Single-View Human Demonstration Retargeting [49.70827726228219]
  We present DemoBridge, a toolkit that turns a single-view RGB stereo recording of a human hand demonstration into an executable, physics-validated robot-arm trajectory.<n>At the core of DemoBridge is a single collision-aware planner.<n>A physics simulator runs in the loop. It validates each phase as it is produced and backtracks on failure, so a demonstration that cannot be reproduced as given is re-planned rather than discarded.
  arXiv  Detail & Related papers  (2026-07-10T15:31:04Z)
• Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models [95.75234389806654]
  We present Qwen-RobotManip, a generalizable Vision-Language-Action foundation model built on Qwen-VL.<n>Qwen-RobotManip introduces a unified alignment framework across the representation, motion, and behavioral dimensions of manipulation.<n>A human-to-robot synthesis pipeline converts egocentric hand demonstrations into robot trajectories across 15 platforms.
  arXiv  Detail & Related papers  (2026-06-16T12:14:39Z)
• Diffusion Transformer World-Action Model for AV Scene Prediction [0.0]
  Action-conditioned world models let an autonomous vehicle predict future camera scenes from its own planned controls.<n>But at compact, trainable scale the futures are ambiguous and the field's standard distortion metrics actively mislead.<n>We confront this with a compact latent world model that predicts future scene latents a frozen decoder renders to $256 times 256$ frames up to 8 seconds ahead.
  arXiv  Detail & Related papers  (2026-06-11T07:24:12Z)
• Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration [19.192199380267393]
  We introduce TouchSafeBench, a benchmark for evaluating collision grounding in vision-language models (VLMs)<n>TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social navigation and social rearrangement.<n>We study two deployment-facing tasks: classifying the current safety state and warning about imminent collision before contact.
  arXiv  Detail & Related papers  (2026-05-29T12:04:38Z)
• Fiducial Exoskeletons: Image-Centric Robot State Estimation [21.491677821308688]
  We introduce Fiducial Exoskeletons, an image-based reformulation of 3D robot state estimation.<n>Our key insight is twofold. First, we cast robot state estimation as 6D pose estimation of each link from a single RGB image.<n>Second, we make per-link 6D pose estimation robust and simple - even without learning.
  arXiv  Detail & Related papers  (2026-01-12T22:04:25Z)
• RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training [27.63332596592781]
  Vision-based pose estimation of articulated robots with unknown joint angles has applications in collaborative robotics and human-robot interaction tasks.<n>Current frameworks use neural network encoders to extract image features and downstream layers to predict joint angles and robot pose.<n>We introduce RoboPEPP, a method that fuses information about the robot's physical model into the encoder using a masking-based self-supervised embedding-predictive architecture.
  arXiv  Detail & Related papers  (2024-11-26T18:26:17Z)
• Exploring 3D Human Pose Estimation and Forecasting from the Robot's Perspective: The HARPER Dataset [52.22758311559]
  We introduce HARPER, a novel dataset for 3D body pose estimation and forecast in dyadic interactions between users and Spot. The key-novelty is the focus on the robot's perspective, i.e., on the data captured by the robot's sensors. The scenario underlying HARPER includes 15 actions, of which 10 involve physical contact between the robot and users.
  arXiv  Detail & Related papers  (2024-03-21T14:53:50Z)
• Real-time Holistic Robot Pose Estimation with Unknown States [30.41806081818826]
  Estimating robot pose from RGB images is a crucial problem in computer vision and robotics. Previous methods presume full knowledge of robot internal states, e.g. ground-truth robot joint angles. This work introduces an efficient framework for real-time robot pose estimation from RGB images without requiring known robot states.
  arXiv  Detail & Related papers  (2024-02-08T13:12:50Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.