OmniAI: A Surface-Adaptive Aerial Projection Interface for Human--Drone Interaction

• URL: http://arxiv.org/abs/2608.00721v1
• Date: Sat, 01 Aug 2026 15:41:36 GMT
• Title: OmniAI: A Surface-Adaptive Aerial Projection Interface for Human--Drone Interaction
• Abstract summary: We present OmniAI, an aerial agent that supports surface-adaptive interaction by switching projection between an onboard screen and nearby surfaces.<n>Projection surfaces are detected online using RGB-D sensing and RANSAC plane fitting, without pre-mapped geometry.
• Score: 2.6822193190929444
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Drones in human environments often lack spatially grounded in- terfaces for situated communication. We present OmniAI, an em- bodied aerial agent that supports surface-adaptive interaction by switching projection between an onboard screen and nearby en- vironmental surfaces. A servo-actuated MEMS laser projector renders text-and-image responses from a web-augmented LLM pipeline. Projection surfaces are detected online using RGB-D sensing and RANSAC plane fitting, without pre-mapped geometry. OmniAI provides functionally equivalent voice and gesture con- trol for both drone motion and projected content. By combining speech, mid-air gestures, adaptive projection, and aerial mobility, OmniAI demonstrates a mobile spatial AR interface for context- aware human-drone interaction.

Related papers

• TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model [49.03909082056646]
  We study the problem of navigating cluttered indoor environments with a humanoid robot.<n>We introduce TANGO, the first whole-body vision-language navigation framework.<n>In simulation experiments, TANGO demonstrates state-of-the-art performance in vision-language navigation.
  arXiv  Detail & Related papers  (2026-09-08T17:59:55Z)
• HybridSim: A Physics-Learning Hybrid Digital Twin for mmWave Human Sensing [16.25180983132509]
  HybridSim is a physics-learning hybrid simulator that synthesizes mmWave radar signals from dynamic human meshes under a fixed indoor room configuration.<n>We show improved agreement with a physically based reference and consistent gains on downstream radar-based human sensing tasks when using HybridSim for site-specific data augmentation.
  arXiv  Detail & Related papers  (2026-07-17T10:20:31Z)
• DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Images [50.96240625673857]
  DroneFINE is a novel PEFT paradigm comprising two domain-aware complementary modules tailored for VLM-based drone image detectors.<n>DroneFINE significantly outperforms existing PEFT methods and achieves performance comparable to full fine-tuning.
  arXiv  Detail & Related papers  (2026-07-01T02:27:40Z)
• OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction [41.5123936517904]
  UMI-style interfaces enable scalable robot learning, but existing systems remain largely visuomotor.<n>We present OmniUMI, a unified framework for physically grounded robot learning via human-aligned multimodal interaction.
  arXiv  Detail & Related papers  (2026-04-12T13:48:48Z)
• Direct Contact-Tolerant Motion Planning With Vision Language Models [64.48049143625063]
  Navigation in cluttered environments often requires robots to tolerate contact with movable or deformable objects.<n>We propose a direct contact-tolerant (DCT) planner, which integrates vision-language models (VLMs) into direct point perception and navigation.<n>We implement DCT in Isaac Sim and a real car-like robot, demonstrating that DCT achieves robust and efficient navigation in cluttered environments with movable obstacles.
  arXiv  Detail & Related papers  (2026-03-05T10:05:00Z)
• MRPoS: Mixed Reality-Based Robot Navigation Interface Using Spatial Pointing and Speech with Large Language Model [1.552282932199974]
  This paper proposes the Mixed Reality-Based Robot Navigation Interface using Spatial Pointing and Speech (MRPoS)<n>This novel framework replaces complex hand gestures with a natural, multimodal interface combining spatial pointing with Large Language Model (LLM)-based speech interaction.<n>By leveraging both information, the system translates verbal intent into navigation goals visualized by MR technology.
  arXiv  Detail & Related papers  (2026-03-04T08:45:25Z)
• SARAH: Spatially Aware Real-time Agentic Humans [58.32612596034656]
  We present the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset.<n>Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user.<n>We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment.
  arXiv  Detail & Related papers  (2026-02-20T18:59:35Z)
• DroneVLA: VLA based Aerial Manipulation [2.1645011609137295]
  This work introduces a novel concept of autonomous aerial manipulation system capable of interpreting high-level natural language commands to retrieve objects and deliver them to a human user.<n>The system is intended to integrate a MediaPipe based on Grounding DINO and a Vision-Language-Action model with a custom-built drone equipped with a 1-DOF gripper and an Intel RealSense RGB-D camera.<n>We demonstrate the system's efficacy through real-world experiments for localization and navigation, which resulted in a 0.164m, 0.070m, and 0.084m of max, mean euclidean, and root-mean squared
  arXiv  Detail & Related papers  (2026-01-20T10:08:00Z)
• HoverAI: An Embodied Aerial Agent for Natural Human-Drone Interaction [2.6822193190929444]
  We present HoverAI, an embodied aerial agent that integrates drone mobility, infrastructure-independent visual projection, and real-time conversational AI into a unified platform.<n>HoverAI perceives users through vision and voice, responding via lip-synced avatars that adapt appearance to user demographics.
  arXiv  Detail & Related papers  (2026-01-20T09:59:49Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction [61.833152949826946]
  We propose a novel Ground-aware Motion Model for 3D Human Motion Reconstruction, named GraMMaR. GraMMaR learns the distribution of transitions in both pose and interaction between every joint and ground plane at each time step of a motion sequence. It is trained to explicitly promote consistency between the motion and distance change towards the ground.
  arXiv  Detail & Related papers  (2023-06-29T07:22:20Z)
• iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes [54.04456391489063]
  iGibson is a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes. Our environment contains fifteen fully interactive home-sized scenes populated with rigid and articulated objects. iGibson features enable the generalization of navigation agents, and that the human-iGibson interface and integrated motion planners facilitate efficient imitation learning of simple human demonstrated behaviors.
  arXiv  Detail & Related papers  (2020-12-05T02:14:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.