ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality

• URL: http://arxiv.org/abs/2604.04905v1
• Date: Mon, 06 Apr 2026 17:50:47 GMT
• Title: ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality
• Abstract summary: ClickAIXR is a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR)<n>It integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm.<n>We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction.
• Score: 45.71425751140069
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present ClickAIXR, a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural language questions through both text and speech. This object-centered interaction reduces ambiguity inherent in gaze- or voice-only interfaces and improves transparency by performing all inference on-device, addressing concerns around privacy and latency. We implemented ClickAIXR in the Magic Leap SDK (C API) with ONNX-based local VLM inference. We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction. Results show that latency is moderate and user experience is acceptable. Our findings demonstrate the potential of click-based object selection combined with on-device AI to advance trustworthy, privacy-preserving XR interactions. The source code and supplementary materials are available at: nanovis.org/ClickAIXR.html

Related papers

• JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence [53.427969116091184]
  We release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model.<n>It makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model.<n>It is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.
  arXiv  Detail & Related papers  (2026-06-10T03:43:50Z)
• Machine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments [2.6880847615603236]
  This dissertation proposes novel architectures to improve intelligent agents across three key vision-language tasks.<n>First, we address limitations in visual representation for image captioning.<n>Second, we tackle visual dialog, which requires multi-turn conversation about an image.<n>Third, we study interactive instruction-following for embodied AI using the ALFRED dataset.
  arXiv  Detail & Related papers  (2026-05-20T06:11:25Z)
• Towards Unconstrained Human-Object Interaction [55.123145316966635]
  Human-Object Interaction (HOI) detection is a longstanding computer vision problem.<n>With the advent of Multimodal Large Language Models (MLLMs) it has become feasible to explore more flexible paradigms for interaction recognition.<n>In this work, we revisit HOI detection through the lens of MLLMs and apply them to in-the-wild HOI detection.
  arXiv  Detail & Related papers  (2026-04-15T16:39:28Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• PRISM-XR: Empowering Privacy-Aware XR Collaboration with Multimodal Large Language Models [8.808170696228865]
  PRISM-XR is a novel framework that facilitates multi-user collaboration in XR environments by providing privacy-aware MLLM integration.<n>Our numerical evaluation results indicate that the proposed platform achieves nearly 90% accuracy in fulfilling user requests.
  arXiv  Detail & Related papers  (2026-02-09T21:28:02Z)
• Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation [6.631248829195371]
  We introduce Hear-Your-Click, an interactive V2A framework enabling users to generate sounds for specific objects by clicking on the frame.<n>To achieve this, we propose Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) with a Mask-guided Visual (MVE) to obtain object-level visual features aligned with audio.<n>To measure audio-visual correspondence, we designed a new evaluation metric, the CAV score.
  arXiv  Detail & Related papers  (2025-07-07T13:01:50Z)
• AIvaluateXR: An Evaluation Framework for on-Device AI in XR with Benchmarking Results [55.33807002543901]
  We present AIvaluateXR, a comprehensive evaluation framework for benchmarking large language models (LLMs) running on XR devices.<n>We deploy 17 selected LLMs across four XR platforms: Magic Leap 2, Meta Quest 3, Vivo X100s Pro, and Apple Vision Pro, and conduct an extensive evaluation.<n>We propose a unified evaluation method based on the 3D Optimality theory to select the optimal device-model pairs from quality and speed objectives.
  arXiv  Detail & Related papers  (2025-02-13T20:55:48Z)
• Spatio-Temporal Context Prompting for Zero-Shot Action Detection [13.22912547389941]
  We propose a method which can effectively leverage the rich knowledge of visual-language models to perform Person-Context Interaction.<n>To address the challenge of recognizing distinct actions by multiple people at the same timestamp, we design the Interest Token Spotting mechanism.<n>Our method achieves superior results compared to previous approaches and can be further extended to multi-action videos.
  arXiv  Detail & Related papers  (2024-08-28T17:59:05Z)
• Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces [1.3107174618549584]
  Instruction Visual Grounding (IVG) is a multi-modal approach to object identification within a Graphical User Interface (GUI)<n>We propose IVGocr, which combines a Large Language Model (LLM), an object detection model, and an Optical Character Recognition (OCR) module; and IVGdirect, which uses a multimodal architecture for end-to-end grounding.<n>Our final test dataset is publicly released to support future research.
  arXiv  Detail & Related papers  (2024-05-05T19:10:19Z)
• Revisiting Click-based Interactive Video Object Segmentation [24.114405100879278]
  CiVOS builds on de-coupled modules reflecting user interaction and mask propagation. The approach is extensively evaluated on the popular interactiveDAVIS dataset. The presented CiVOS pipeline achieves competitive results, although requiring a lower user workload.
  arXiv  Detail & Related papers  (2022-03-03T15:55:14Z)
• Multi-modal Transformers Excel at Class-agnostic Object Detection [105.10403103027306]
  We argue that existing methods lack a top-down supervision signal governed by human-understandable semantics. We develop an efficient and flexible MViT architecture using multi-scale feature processing and deformable self-attention. We show the significance of MViT proposals in a diverse range of applications.
  arXiv  Detail & Related papers  (2021-11-22T18:59:29Z)
• Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion [68.45737688496654]
  We present a modular interactive VOS framework which decouples interaction-to-mask and mask propagation. We show that our method outperforms current state-of-the-art algorithms while requiring fewer frame interactions.
  arXiv  Detail & Related papers  (2021-03-14T14:39:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.