DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics

• URL: http://arxiv.org/abs/2312.06408v1
• Date: Mon, 11 Dec 2023 14:29:25 GMT
• Title: DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
• Authors: Zhiao Huang, Feng Chen, Yewen Pu, Chunru Lin, Hao Su, Chuang Gan
• Abstract summary: DiffVL is a method that enables non-expert users to communicate soft-body manipulation tasks. We leverage large language models to translate task descriptions into machine-interpretable optimization objectives.
• Score: 69.6158232150048
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems. Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory. However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users. We introduce DiffVL, a method that enables non-expert users to communicate soft-body manipulation tasks -- a combination of vision and natural language, given in multiple stages -- that can be readily leveraged by a differential physics solver. We have developed GUI tools that enable non-expert users to specify 100 tasks inspired by real-life soft-body manipulations from online videos, which we'll make public. We leverage large language models to translate task descriptions into machine-interpretable optimization objectives. The optimization objectives can help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines.

Related papers

• Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate [105.86576388991713]
  We introduce a normalized gradient difference (NGDiff) algorithm, enabling us to have better control over the trade-off between the objectives. We provide a theoretical analysis and empirically demonstrate the superior performance of NGDiff among state-of-the-art unlearning methods on the TOFU and MUSE datasets.
  arXiv  Detail & Related papers  (2024-10-29T14:41:44Z)
• Bayesian optimization for robust robotic grasping using a sensorized compliant hand [6.693397171872655]
  We analyze different grasp metrics to provide realistic grasp optimization in a real system including tactile sensors. An experimental evaluation in the robotic system shows the usefulness of the method for performing unknown object grasping.
  arXiv  Detail & Related papers  (2024-10-23T19:33:14Z)
• Deep-ELA: Deep Exploratory Landscape Analysis with Self-Supervised Pretrained Transformers for Single- and Multi-Objective Continuous Optimization Problems [0.5461938536945723]
  In this work, we propose a hybrid approach, Deep-ELA, which combines (the benefits of) deep learning and ELA features. Our proposed framework can either be used out-of-the-box for analyzing single- and multi-objective continuous optimization problems, or subsequently fine-tuned to various tasks.
  arXiv  Detail & Related papers  (2024-01-02T12:41:17Z)
• Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics [99.86385468927638]
  We propose a contact point discovery approach (CPDeform) that guides the differentiable physics solver to deform soft-body plasticines. On single-stage tasks, our method can automatically find suitable initial contact points based on transport priorities. On complex multi-stage tasks, we can iteratively switch the contact points of end-effectors based on transport priorities.
  arXiv  Detail & Related papers  (2022-05-05T17:59:41Z)
• DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools [96.38972082580294]
  DiffSkill is a novel framework that uses a differentiable physics simulator for skill abstraction to solve deformable object manipulation tasks. In particular, we first obtain short-horizon skills using individual tools from a gradient-based simulator. We then learn a neural skill abstractor from the demonstration trajectories which takes RGBD images as input.
  arXiv  Detail & Related papers  (2022-03-31T17:59:38Z)
• How to Fill the Optimum Set? Population Gradient Descent with Harmless Diversity [34.790747999729284]
  We propose a bi-level optimization problem of maximizing a diversity score inside the optimum set of the main loss function. We show that our method can efficiently generate diverse solutions on a variety of applications, including text-to-image generation, text-to-mesh generation, molecular conformation generation and ensemble neural network training.
  arXiv  Detail & Related papers  (2022-02-16T23:40:18Z)
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics [89.81550748680245]
  We introduce a new differentiable physics benchmark called PasticineLab. In each task, the agent uses manipulators to deform the plasticine into the desired configuration. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark.
  arXiv  Detail & Related papers  (2021-04-07T17:59:23Z)
• Scalable Differentiable Physics for Learning and Control [99.4302215142673]
  Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments. We develop a scalable framework for differentiable physics that can support a large number of objects and their interactions.
  arXiv  Detail & Related papers  (2020-07-04T19:07:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.