ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals

• URL: http://arxiv.org/abs/2309.12312v2
• Date: Sun, 24 Sep 2023 01:24:06 GMT
• Title: ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals
• Abstract summary: We present ForceSight, a system for text-guided mobile manipulation that predicts visual-force goals using a deep neural network. ForceSight performed tasks such as precision grasps, drawer opening, and object handovers with an 81% success rate in unseen environments. In a separate experiment, relying exclusively on visual servoing and ignoring force goals dropped the success rate from 90% to 45%, demonstrating that force goals can significantly enhance performance.
• Score: 7.728447960544735
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present ForceSight, a system for text-guided mobile manipulation that predicts visual-force goals using a deep neural network. Given a single RGBD image combined with a text prompt, ForceSight determines a target end-effector pose in the camera frame (kinematic goal) and the associated forces (force goal). Together, these two components form a visual-force goal. Prior work has demonstrated that deep models outputting human-interpretable kinematic goals can enable dexterous manipulation by real robots. Forces are critical to manipulation, yet have typically been relegated to lower-level execution in these systems. When deployed on a mobile manipulator equipped with an eye-in-hand RGBD camera, ForceSight performed tasks such as precision grasps, drawer opening, and object handovers with an 81% success rate in unseen environments with object instances that differed significantly from the training data. In a separate experiment, relying exclusively on visual servoing and ignoring force goals dropped the success rate from 90% to 45%, demonstrating that force goals can significantly enhance performance. The appendix, videos, code, and trained models are available at https://force-sight.github.io/.

Related papers

• ReForce: Learning Force-aware Retargeting for Dexterous Manipulation [9.525202879335618]
  ReForce is a Force-aware Retargeting method that turns human motion and forces into robot actions that reproduce the intended contact.<n>It supports both online force-aware teleoperation and offline data translation.<n>It achieves lower force-tracking error and stronger multi-finger contact engagement on contact-rich tasks such as paper-cup grasping and tongs manipulation.
  arXiv  Detail & Related papers  (2026-08-16T06:13:14Z)
• ForceBand: Learning Forceful Manipulation with sEMG [55.186900419710746]
  We introduce ForceBand, a low-cost wrist-worn sEMG system that turns human muscle activity into force-enriched demonstrations.<n>We pre-train an EMG2Force model that predicts per-finger forces from sEMG and IMU signals.<n>Experiments show that ForceBand recovers fine-grained fingertip interactions with over 50% lower force prediction error than vision-based baselines.
  arXiv  Detail & Related papers  (2026-06-24T17:59:35Z)
• Contrastive Action-Image Pre-training for Visuomotor Control [91.5630572541332]
  We introduce CAIP, a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions.<n>Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders.
  arXiv  Detail & Related papers  (2026-06-15T20:00:20Z)
• VOFA: Visual Object Goal Pushing with Force-Adaptive Control for Humanoids [35.671376073584305]
  VOFA is a visual goal-conditioned humanoid loco-manipulation system.<n>It is capable of pushing objects with unknown physical properties to arbitrary goal positions.<n>VOFA is extensively evaluated in both simulation and real-world experiments on the Booster T1 humanoid robot.
  arXiv  Detail & Related papers  (2026-05-02T16:16:23Z)
• Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation [14.013652439013692]
  This paper presents a new paradigm, HERO, for object loco-manipulation with humanoid robots.<n>We achieve this by designing an accurate residual-aware EE tracking policy.<n>We use this accurate end-effector tracker to build a modular system for loco-manipulation.
  arXiv  Detail & Related papers  (2026-02-18T18:55:02Z)
• Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction [24.49384094440561]
  We propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human videos.<n>In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand.<n>In the real world, manipulation policies trained from RGB videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand.
  arXiv  Detail & Related papers  (2026-02-09T18:56:02Z)
• Feel the Force: Contact-Driven Learning from Humans [52.36160086934298]
  Controlling fine-grained forces during manipulation remains a core challenge in robotics.<n>We present FeelTheForce, a robot learning system that models human tactile behavior to learn force-sensitive manipulation.<n>Our approach grounds robust low-level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks.
  arXiv  Detail & Related papers  (2025-06-02T17:57:52Z)
• Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions [37.334138196925025]
  Functional grasp is essential for enabling dexterous multi-finger robot hands to manipulate objects effectively.<n>We propose extracting human grasp information from web images since they depict natural and functional object interactions.<n>We show that these relatively low-quality HOI data from inexpensive web sources can effectively train a functional grasping model.
  arXiv  Detail & Related papers  (2025-05-07T16:13:17Z)
• ForceGrip: Reference-Free Curriculum Learning for Realistic Grip Force Control in VR Hand Manipulation [0.10995326465245926]
  We present ForceGrip, a deep learning agent that synthesizes realistic hand manipulation motions. We employ a three-phase curriculum learning framework comprising Finger Positioning, Intention Adaptation, and Dynamic Stabilization. Our evaluations reveal ForceGrip's superior force controllability and plausibility compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-11T05:39:07Z)
• A Training-Free Framework for Precise Mobile Manipulation of Small Everyday Objects [16.018172627950857]
  We develop a closed-loop training-free framework that enables a mobile manipulator to tackle precise tasks involving the manipulation of small objects. SVM employs an RGB-D wrist camera and uses visual servoing for control. We demonstrate that open-vocabulary object detectors can serve as a drop-in module to identify semantic targets.
  arXiv  Detail & Related papers  (2025-02-19T18:59:17Z)
• OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation [35.97702591413093]
  We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately.
  arXiv  Detail & Related papers  (2024-10-15T17:17:54Z)
• Hand-Object Interaction Pretraining from Videos [77.92637809322231]
  We learn general robot manipulation priors from 3D hand-object interaction trajectories. We do so by sharing both the human hand and the manipulated object in 3D space and human motions to robot actions. We empirically demonstrate that finetuning this policy, with both reinforcement learning (RL) and behavior cloning (BC), enables sample-efficient adaptation to downstream tasks and simultaneously improves robustness and generalizability compared to prior approaches.
  arXiv  Detail & Related papers  (2024-09-12T17:59:07Z)
• Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models [53.22792173053473]
  We introduce an interactive robotic manipulation framework called Polaris. Polaris integrates perception and interaction by utilizing GPT-4 alongside grounded vision models. We propose a novel Synthetic-to-Real (Syn2Real) pose estimation pipeline.
  arXiv  Detail & Related papers  (2024-08-15T06:40:38Z)
• Click to Grasp: Zero-Shot Precise Manipulation via Visual Diffusion Descriptors [30.579707929061026]
  Our work explores the grounding of fine-grained part descriptors for precise manipulation in a zero-shot setting. We tackle the problem by framing it as a dense semantic part correspondence task. Our model returns a gripper pose for manipulating a specific part, using as reference a user-defined click from a source image of a visually different instance of the same object.
  arXiv  Detail & Related papers  (2024-03-21T16:26:19Z)
• Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation [57.60490773016364]
  We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation.
  arXiv  Detail & Related papers  (2023-12-20T22:36:37Z)
• What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [57.92924256181857]
  We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
  arXiv  Detail & Related papers  (2023-11-03T18:09:08Z)
• CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesis [18.618218866876422]
  We present a novel task of category-level functional hand-object manipulation synthesis. We generate a physically reasonable hand-object manipulation sequence that performs like human beings. Our framework achieves state-of-the-art performance for both rigid and articulated categories with impressive visual effects.
  arXiv  Detail & Related papers  (2023-03-25T07:03:12Z)
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? [54.442692221567796]
  Task specification is critical for engagement of non-expert end-users and adoption of personalized robots. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
  arXiv  Detail & Related papers  (2022-04-23T19:39:49Z)
• Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects [79.351446087227]
  We address the problem of inferring contact points and the physical forces from videos of humans interacting with objects. Specifically, we use a simulator to predict effects and enforce that estimated forces must lead to the same effect as depicted in the video.
  arXiv  Detail & Related papers  (2020-03-26T17:20:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.