Which architecture should be implemented to manage data from the real world, in an Unreal Engine 5 simulator and in the context of mixed reality?

• URL: http://arxiv.org/abs/2305.09244v1
• Date: Tue, 16 May 2023 07:51:54 GMT
• Title: Which architecture should be implemented to manage data from the real world, in an Unreal Engine 5 simulator and in the context of mixed reality?
• Authors: Jonathan Cassaing
• Abstract summary: This paper gives a detailed analysis of the issue, at both theoretical and operational level. The C++ system is reviewed in details as well as the third-party library: pitfalls to be avoided are shown. The last chapter proposes a generic architecture, useful in large-scale industrial 3D applications.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to its ability to generate millions of particles, massively detailed scenes and confusing artificial illumination with reality, the version 5 of Unreal Engine promises unprecedented industrial applications. The paradigms and aims of Unreal Engine contrast with the industrial simulators typically used by the scientific community. The visual quality and performance of its rendering engine increase the opportunities, especially for industries and simulation business: where interoperability and scalability are required. The study of the following issue `` Which architecture should we implement to integrate real-world data, in an Unreal Engine 5 simulator and in a mixed-reality environment? '' offers a point of view. The topic is reexamined in an innovative and conceptual way, such as the generalization of mixedreality technologies, Internet of Things, digital twins, Big Data but providing a solution for simple and actual use cases. This paper gives a detailed analysis of the issue, at both theoretical and operational level. Then, the document goes deep into Unreal Engine's operation in order to extract the vanilla capabilities. Next, the C++ Plugin system is reviewed in details as well as the third-party library integration: pitfalls to be avoided are shown. Finally, the last chapter proposes a generic architecture, useful in large-scale industrial 3D applications, such as collaborative work or hyper-connected simulators. This document might be of interest to an Unreal Engine expert who would like to discover about server architectures. Conversely, it could be relevant for an expert in backend servers who wants to learn about Unreal Engine capabilities. This research concludes that Unreal Engine's modularity enables integration with almost any protocol. The features to integrate external real data are numerous but depend on use cases. Distributed systems for Big Data require a scalable architecture, possibly without the use of the Unreal Engine dedicated server. Environments, which require sub-second latency need to implement direct connections, bypassing any intermediate servers.

Related papers

• Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation [16.69742672616517]
  We introduce an innovative structured light simulation system, generating both RGB and physically realistic depth images. We create an RGBD dataset tailored for robotic industrial grasping scenarios. By reducing the sim2real gap and enhancing deep learning training, we facilitate the application of deep learning models in industrial settings.
  arXiv  Detail & Related papers  (2024-07-17T09:57:14Z)
• AgentStudio: A Toolkit for Building General Virtual Agents [57.02375267926862]
  We introduce AgentStudio, an online, realistic, and multimodal toolkit that covers the entire lifecycle of agent development. This includes environment setups, data collection, agent evaluation, and visualization. We have open-sourced the environments, datasets, benchmarks, and interfaces to promote research towards developing general virtual agents.
  arXiv  Detail & Related papers  (2024-03-26T17:54:15Z)
• MuSHRoom: Multi-Sensor Hybrid Room Dataset for Joint 3D Reconstruction and Novel View Synthesis [26.710960922302124]
  We propose a real-world Multi-Sensor Hybrid Room dataset (MuSHRoom) Our dataset presents exciting challenges and requires state-of-the-art methods to be cost-effective, robust to noisy data and devices. We benchmark several famous pipelines on our dataset for joint 3D mesh reconstruction and novel view synthesis.
  arXiv  Detail & Related papers  (2023-11-05T21:46:12Z)
• Sim2real Transfer Learning for Point Cloud Segmentation: An Industrial Application Case on Autonomous Disassembly [55.41644538483948]
  We present an industrial application case that uses sim2real transfer learning for point cloud data. We provide insights on how to generate and process synthetic point cloud data. A novel patch-based attention network is proposed additionally to tackle this problem.
  arXiv  Detail & Related papers  (2023-01-12T14:00:37Z)
• Kubric: A scalable dataset generator [73.78485189435729]
  Kubric is a Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation.
  arXiv  Detail & Related papers  (2022-03-07T18:13:59Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis [78.26022688167133]
  We present a large-scale benchmark dataset for vision-driven robotic grasping via physics-based metaverse synthesis. The proposed dataset contains 100,000 images and 25 different object types. We also propose a new layout-weighted performance metric alongside the dataset for evaluating object detection and segmentation performance.
  arXiv  Detail & Related papers  (2021-12-29T17:23:24Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation [98.51313127382937]
  We focus on the use of labels in the synthetic domain alone. Our approach introduces both a way to learn neural-invariant representations and a theoretically inspired view on how to sample the data from the simulator. We showcase our approach on the bird's-eye-view vehicle segmentation task with multi-sensor data.
  arXiv  Detail & Related papers  (2021-11-15T18:37:43Z)
• UnrealROX+: An Improved Tool for Acquiring Synthetic Data from Virtual 3D Environments [14.453602631430508]
  We present an improved version of UnrealROX, a tool to generate synthetic data from robotic images. Un UnrealROX+ includes new features such as generating albedo or a Python API for interacting with the virtual environment from Deep Learning frameworks.
  arXiv  Detail & Related papers  (2021-04-23T18:45:42Z)
• SAILenv: Learning in Virtual Visual Environments Made Simple [16.979621213790015]
  We present a novel platform that allows researchers to experiment visual recognition in virtual 3D scenes. A few lines of code are needed to interface every algorithm with the virtual world, and non-3D-graphics experts can easily customize the 3D environment itself. Our framework yields pixel-level semantic and instance labeling, depth, and, to the best of our knowledge, it is the only one that provides motion-related information directly inherited from the 3D engine.
  arXiv  Detail & Related papers  (2020-07-16T09:50:23Z)
• Tidying Deep Saliency Prediction Architectures [6.613005108411055]
  In this paper, we identify four key components of saliency models, i.e., input features, multi-level integration, readout architecture, and loss functions. We propose two novel end-to-end architectures called SimpleNet and MDNSal, which are neater, minimal, more interpretable and achieve state of the art performance on public saliency benchmarks.
  arXiv  Detail & Related papers  (2020-03-10T19:34:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.