Social Conjuring: Multi-User Runtime Collaboration with AI in Building Virtual 3D Worlds

• URL: http://arxiv.org/abs/2410.00274v2
• Date: Wed, 2 Oct 2024 17:34:41 GMT
• Title: Social Conjuring: Multi-User Runtime Collaboration with AI in Building Virtual 3D Worlds
• Authors: Amina Kobenova, Cyan DeVeaux, Samyak Parajuli, Andrzej Banburski-Fahey, Judith Amores Fernandez, Jaron Lanier,
• Abstract summary: Social Conjurer is a framework for AI-augmented dynamic 3D scene co-creation. This article presents a set of implications for designing human-centered interfaces that incorporate AI models into 3D content generation.
• Score: 3.5152339192019113
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative artificial intelligence has shown promise in prompting virtual worlds into existence, yet little attention has been given to understanding how this process unfolds as social interaction. We present Social Conjurer, a framework for AI-augmented dynamic 3D scene co-creation, where multiple users collaboratively build and modify virtual worlds in real-time. Through an expanded set of interactions, including social and tool-based engagements as well as spatial reasoning, our framework facilitates the creation of rich, diverse virtual environments. Findings from a preliminary user study (N=12) provide insight into the user experience of this approach, how social contexts shape the prompting of spatial environments, and perspective on social applications of prompt-based 3D co-creation. In addition to highlighting the potential of AI-supported multi-user world creation and offering new pathways for AI-augmented creative processes in VR, this article presents a set of implications for designing human-centered interfaces that incorporate AI models into 3D content generation.

Related papers

• Generative AI for Accessible and Inclusive Extended Reality [20.669785157017486]
  We discuss potential benefits but also challenges that AIGC has for the creation of inclusive and accessible virtual environments. Specifically, we touch upon the decreased need for 3D modeling expertise, benefits of symbolic-only as well as multimodal input, 3D content editing, and 3D model accessibility as well as foundation model-specific challenges.
  arXiv  Detail & Related papers  (2024-10-31T10:43:43Z)
• Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI [10.335943413484815]
  seamless integration of virtual and physical worlds in augmented reality benefits from the system semantically "understanding" the physical environment. We introduce a multimodal 3D object representation that unifies both semantic and linguistic knowledge with the geometric representation. We demonstrate the usefulness of the proposed system through two real-world AR applications on Magic Leap 2: a) spatial search in physical environments with natural language and b) an intelligent inventory system that tracks object changes over time.
  arXiv  Detail & Related papers  (2024-10-06T23:25:21Z)
• SpaceBlender: Creating Context-Rich Collaborative Spaces Through Generative 3D Scene Blending [19.06858242647237]
  We introduce SpaceBlender, a pipeline that transforms user-provided 2D images into context-rich 3D environments. Participants appreciated the enhanced familiarity and context provided by SpaceBlender but noted complexities in the generative environments. We propose directions for improving the pipeline and discuss the value and design of blended spaces for different scenarios.
  arXiv  Detail & Related papers  (2024-09-20T22:27:31Z)
• On the Emergence of Symmetrical Reality [51.21203247240322]
  We introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. We propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality.
  arXiv  Detail & Related papers  (2024-01-26T16:09:39Z)
• Agent AI: Surveying the Horizons of Multimodal Interaction [83.18367129924997]
  "Agent AI" is a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data. We envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.
  arXiv  Detail & Related papers  (2024-01-07T19:11:18Z)
• Beyond Reality: The Pivotal Role of Generative AI in the Metaverse [98.1561456565877]
  This paper offers a comprehensive exploration of how generative AI technologies are shaping the Metaverse. We delve into the applications of text generation models like ChatGPT and GPT-3, which are enhancing conversational interfaces with AI-generated characters. We also examine the potential of 3D model generation technologies like Point-E and Lumirithmic in creating realistic virtual objects.
  arXiv  Detail & Related papers  (2023-07-28T05:44:20Z)
• ArK: Augmented Reality with Knowledge Interactive Emergent Ability [115.72679420999535]
  We develop an infinite agent that learns to transfer knowledge memory from general foundation models to novel domains. The heart of our approach is an emerging mechanism, dubbed Augmented Reality with Knowledge Inference Interaction (ArK) We show that our ArK approach, combined with large foundation models, significantly improves the quality of generated 2D/3D scenes.
  arXiv  Detail & Related papers  (2023-05-01T17:57:01Z)
• Artificial Intelligence for the Metaverse: A Survey [66.57225253532748]
  We first deliver a preliminary of AI, including machine learning algorithms and deep learning architectures, and its role in the metaverse. We then convey a comprehensive investigation of AI-based methods concerning six technical aspects that have potentials for the metaverse. Several AI-aided applications, such as healthcare, manufacturing, smart cities, and gaming, are studied to be deployed in the virtual worlds.
  arXiv  Detail & Related papers  (2022-02-15T03:34:56Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.