Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation

• URL: http://arxiv.org/abs/2502.02201v1
• Date: Tue, 04 Feb 2025 10:27:40 GMT
• Title: Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation
• Authors: Xiangzhi Eric Wang, Zackary P. T. Sin, Ye Jia, Daniel Archer, Wynonna H. Y. Fong, Qing Li, Chen Li,
• Abstract summary: We propose VR Mover, an LLM-empowered solution that can understand and interpret the user's vocal instruction to support object manipulation. Our user study demonstrates that VR Mover enhances user usability, overall experience and performance on multi-object manipulation.
• Score: 12.569646616546235
• License:
• Abstract: In our daily lives, we can naturally convey instructions for the spatial manipulation of objects using words and gestures. Transposing this form of interaction into virtual reality (VR) object manipulation can be beneficial. We propose VR Mover, an LLM-empowered solution that can understand and interpret the user's vocal instruction to support object manipulation. By simply pointing and speaking, the LLM can manipulate objects without structured input. Our user study demonstrates that VR Mover enhances user usability, overall experience and performance on multi-object manipulation, while also reducing workload and arm fatigue. Users prefer the proposed natural interface for broad movements and may complementarily switch to gizmos or virtual hands for finer adjustments. These findings are believed to contribute to design implications for future LLM-based object manipulation interfaces, highlighting the potential for more intuitive and efficient user interactions in VR environments.

Related papers

• VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications [2.5022287664959446]
  This study introduces a pioneering approach utilizing Visual Language Models within VR environments to enhance user interaction and task efficiency. Our system facilitates real-time, intuitive user interactions through natural language processing, without relying on visual text instructions.
  arXiv  Detail & Related papers  (2024-05-19T12:56:00Z)
• Tremor Reduction for Accessible Ray Based Interaction in VR Applications [0.0]
  Many traditional 2D interface interaction methods have been directly converted to work in a VR space with little alteration to the input mechanism. In this paper we propose the use of a low pass filter, to normalize user input noise, alleviating fine motor requirements during ray-based interaction.
  arXiv  Detail & Related papers  (2024-05-12T17:07:16Z)
• Learning High-Quality Navigation and Zooming on Omnidirectional Images in Virtual Reality [37.564863636844905]
  We present a novel system, called OmniVR, designed to enhance visual clarity during VR navigation. Our system enables users to effortlessly locate and zoom in on the objects of interest in VR.
  arXiv  Detail & Related papers  (2024-05-01T07:08:24Z)
• Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs [53.66070434419739]
  Generalizable articulated object manipulation is essential for home-assistant robots. We propose a kinematic-aware prompting framework that prompts Large Language Models with kinematic knowledge of objects to generate low-level motion waypoints. Our framework outperforms traditional methods on 8 categories seen and shows a powerful zero-shot capability for 8 unseen articulated object categories.
  arXiv  Detail & Related papers  (2023-11-06T03:26:41Z)
• Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
  We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator.
  arXiv  Detail & Related papers  (2023-10-06T20:48:43Z)
• CALM: Conditional Adversarial Latent Models for Directable Virtual Characters [71.66218592749448]
  We present Conditional Adversarial Latent Models (CALM), an approach for generating diverse and directable behaviors for user-controlled interactive virtual characters. Using imitation learning, CALM learns a representation of movement that captures the complexity of human motion, and enables direct control over character movements.
  arXiv  Detail & Related papers  (2023-05-02T09:01:44Z)
• Privacy concerns from variances in spatial navigability in VR [0.0]
  Current Virtual Reality (VR) input devices make it possible to navigate a virtual environment and record immersive, personalized data regarding the user's movement and specific behavioral habits. In this article, the authors propose to investigate Machine Learning driven learning algorithms that try to learn with human users co-operatively and can be used to countermand existing privacy concerns in VR.
  arXiv  Detail & Related papers  (2023-02-06T01:48:59Z)
• Force-Aware Interface via Electromyography for Natural VR/AR Interaction [69.1332992637271]
  We design a learning-based neural interface for natural and intuitive force inputs in VR/AR. We show that our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. We envision our findings to push forward research towards more realistic physicality in future VR/AR.
  arXiv  Detail & Related papers  (2022-10-03T20:51:25Z)
• The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments [81.5101473684021]
  This work proposes a hand gesture authoring tool for object specific grab gestures allowing virtual objects to be grabbed as in the real world. The presented solution uses template matching for gesture recognition and requires no technical knowledge to design and create custom tailored hand gestures. The study showed that gestures created with the proposed approach are perceived by users as a more natural input modality than the others.
  arXiv  Detail & Related papers  (2022-07-03T18:33:33Z)
• Learning Effect of Lay People in Gesture-Based Locomotion in Virtual Reality [81.5101473684021]
  Some of the most promising methods are gesture-based and do not require additional handheld hardware. Recent work focused mostly on user preference and performance of the different locomotion techniques. This work is investigated whether and how quickly users can adapt to a hand gesture-based locomotion system in VR.
  arXiv  Detail & Related papers  (2022-06-16T10:44:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.