HoloLens 2 Research Mode as a Tool for Computer Vision Research

• URL: http://arxiv.org/abs/2008.11239v1
• Date: Tue, 25 Aug 2020 19:05:38 GMT
• Title: HoloLens 2 Research Mode as a Tool for Computer Vision Research
• Abstract summary: We present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications. We also show how to combine the Research Mode sensor data with the built-in eye and hand tracking capabilities.
• Score: 45.404600947276826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with integrated compute capabilities, which makes it an ideal platform for computer vision research. In this technical report, we present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications based on processing sensor data. We also show how to combine the Research Mode sensor data with the built-in eye and hand tracking capabilities provided by HoloLens 2. By releasing the Research Mode API and a set of open-source tools, we aim to foster further research in the fields of computer vision as well as robotics and encourage contributions from the research community.

Related papers

• Ascribe New Dimensions to Scientific Data Visualization with VR [1.9084093324993718]
  This article introduces ASCRIBE-VR, a VR platform of Autonomous Solutions for Computational Research with Immersive Browsing & Exploration. ASCRIBE-VR enables multimodal analysis, structural assessments, and immersive visualization, supporting scientific visualization of advanced datasets such as X-ray CT, Magnetic Resonance, and synthetic 3D imaging.
  arXiv  Detail & Related papers  (2025-04-18T03:59:39Z)
• OpenFly: A Versatile Toolchain and Large-scale Benchmark for Aerial Vision-Language Navigation [49.697035403548966]
  Vision-Language Navigation (VLN) aims to guide agents through an environment by leveraging both language instructions and visual cues. We propose OpenFly, a platform comprising a versatile toolchain and large-scale benchmark for aerial VLN. We construct a large-scale aerial VLN dataset with 100k trajectories, covering diverse heights and lengths across 18 scenes. The corresponding visual data are generated using various rendering engines and advanced techniques, including Unreal, GTA V, Google Earth, and 3D Splatting (3D GS)
  arXiv  Detail & Related papers  (2025-02-25T09:57:18Z)
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs [56.391404083287235]
  We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes.
  arXiv  Detail & Related papers  (2024-06-24T17:59:42Z)
• Detection and Pose Estimation of flat, Texture-less Industry Objects on HoloLens using synthetic Training [0.0]
  Current state-of-the-art 6d pose estimation is too compute intensive to be deployed on edge devices. We propose a synthetically trained client-server-based augmented reality application.
  arXiv  Detail & Related papers  (2024-02-07T15:57:28Z)
• EasyVolcap: Accelerating Neural Volumetric Video Research [69.59671164891725]
  Volumetric video is a technology that digitally records dynamic events such as artistic performances, sporting events, and remote conversations. EasyVolcap is a Python & Pytorch library for unifying the process of multi-view data processing, 4D scene reconstruction, and efficient dynamic volumetric video rendering.
  arXiv  Detail & Related papers  (2023-12-11T17:59:46Z)
• Open Gaze: Open Source eye tracker for smartphone devices using Deep Learning [0.0]
  We present an open-source implementation of a smartphone-based gaze tracker that emulates the methodology proposed by a GooglePaper. Through the integration of machine learning techniques, we unveil an accurate eye tracking solution that is native to smartphones. Our findings exhibit the inherent potential to amplify eye movement research by significant proportions.
  arXiv  Detail & Related papers  (2023-08-25T17:10:22Z)
• Project Aria: A New Tool for Egocentric Multi-Modal AI Research [44.91047009314488]
  Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception. Our team at Meta Reality Labs Research built the Aria device, an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research in this area.
  arXiv  Detail & Related papers  (2023-08-24T20:42:21Z)
• Literature Review: Computer Vision Applications in Transportation Logistics and Warehousing [58.720142291102135]
  Computer vision applications in transportation logistics and warehousing have a huge potential for process automation. We present a structured literature review on research in the field to help leverage this potential.
  arXiv  Detail & Related papers  (2023-04-12T17:33:41Z)
• Low-cost Efficient Wireless Intelligent Sensor (LEWIS) for Engineering, Research, and Education [72.2614468437919]
  The vision of smart cities equipped with sensors informing decisions has not been realized to date. Civil engineers lack of knowledge in sensor technology. The electrical components and computer knowledge associated with sensors are still a challenge for civil engineers.
  arXiv  Detail & Related papers  (2023-03-23T21:49:26Z)
• VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse [77.52012928882928]
  INDOORKIT is a built-in toolkit for NVIDIA OMNIVERSE. It provides flexible pipelines for indoor scene building, scene randomizing, and animation controls.
  arXiv  Detail & Related papers  (2022-06-23T17:53:33Z)
• Efficient Pipelines for Vision-Based Context Sensing [0.24366811507669117]
  There is an emergence of vision sources deployed worldwide. The cameras could be installed on roadside, in-house, and on mobile platforms. However, the vision data collection and analytics are still highly manual today. There are three major challenges for today's vision-based context sensing systems.
  arXiv  Detail & Related papers  (2020-11-01T05:09:13Z)
• A 3D-Deep-Learning-based Augmented Reality Calibration Method for Robotic Environments using Depth Sensor Data [5.027571997864707]
  We propose a novel approach to calibrate the Augmented Reality device using 3D depth sensor data. We use the depth camera of a cutting edge Augmented Reality Device - the Microsoft Hololens for deep learning based calibration. We introduce an open source 3D point cloud labeling tool, which is to our knowledge the first open source tool for labeling raw point cloud data.
  arXiv  Detail & Related papers  (2019-12-27T13:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.