DreamGarden: A Designer Assistant for Growing Games from a Single Prompt

• URL: http://arxiv.org/abs/2410.01791v1
• Date: Wed, 2 Oct 2024 17:49:07 GMT
• Title: DreamGarden: A Designer Assistant for Growing Games from a Single Prompt
• Authors: Sam Earle, Samyak Parajuli, Andrzej Banburski-Fahey,
• Abstract summary: We present DreamGarden, an AI system capable of assisting with the development of diverse game environments in Unreal Engine. At the core of our method is an LLM-driven planner, capable of breaking down a single, high-level prompt into a hierarchical action plan. This system is presented to the user as a garden of plans and actions, both growing independently and responding to user intervention via seed prompts, pruning, and feedback.
• Score: 1.8906710320196731
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coding assistants are increasingly leveraged in game design, both generating code and making high-level plans. To what degree can these tools align with developer workflows, and what new modes of human-computer interaction can emerge from their use? We present DreamGarden, an AI system capable of assisting with the development of diverse game environments in Unreal Engine. At the core of our method is an LLM-driven planner, capable of breaking down a single, high-level prompt -- a dream, memory, or imagined scenario provided by a human user -- into a hierarchical action plan, which is then distributed across specialized submodules facilitating concrete implementation. This system is presented to the user as a garden of plans and actions, both growing independently and responding to user intervention via seed prompts, pruning, and feedback. Through a user study, we explore design implications of this system, charting courses for future work in semi-autonomous assistants and open-ended simulation design.

Related papers

• Multi-Actor Generative Artificial Intelligence as a Game Engine [49.360775442760314]
  Generative AI can be used in multi-actor environments with purposes ranging from social science modeling to interactive narrative and AI evaluation.<n>We argue here that a good approach is to take inspiration from tabletop role-playing games (TTRPGs), where a Game Master (GM) is responsible for the environment and generates all parts of the story not directly determined by the voluntary actions of player characters.
  arXiv  Detail & Related papers  (2025-07-10T22:31:09Z)
• Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI [0.36868085124383626]
  Review presents a comprehensive analysis of two emerging paradigms in AI-assisted software development: vibe coding and agentic coding.<n> Vibe coding emphasizes intuitive, human-in-the-loop interaction through prompt-based, conversational interaction.<n>Agentic coding enables autonomous software development through goal-driven agents capable of planning, executing, testing, and iterating tasks with minimal human intervention.
  arXiv  Detail & Related papers  (2025-05-26T03:00:21Z)
• WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents [67.31920821192323]
  We introduce WorldCraft, a system where large language model (LLM) agents leverage procedural generation to create scenes populated with objects. In our framework, a coordinator agent manages the overall process and works with two specialized LLM agents to complete the scene creation. Our pipeline incorporates a trajectory control agent, allowing users to animate the scene and operate the camera through natural language interactions.
  arXiv  Detail & Related papers  (2025-02-21T17:18:30Z)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
  We present a framework for intuitive robot programming by non-experts. We leverage natural language prompts and contextual information from the Robot Operating System (ROS) Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
  arXiv  Detail & Related papers  (2024-06-28T08:28:38Z)
• Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction [0.40964539027092917]
  Designers often seek to interact with the software in a more intelligent and lightweight manner. We propose an autonomous agent framework that can function as a copilot in the BIM authoring tool. In a case study based on the BIM authoring software Vectorworks, we implemented a software prototype to integrate the proposed framework seamlessly.
  arXiv  Detail & Related papers  (2024-06-02T17:47:57Z)
• Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality [28.27036270001756]
  This work designs an autonomous workflow tailored for integrating AI agents seamlessly into extended reality (XR) applications for fine-grained training. We present a demonstration of a multimodal fine-grained training assistant for LEGO brick assembly in a pilot XR environment.
  arXiv  Detail & Related papers  (2024-05-16T14:20:30Z)
• Scaling Instructable Agents Across Many Simulated Worlds [70.97268311053328]
  Our goal is to develop an agent that can accomplish anything a human can do in any simulated 3D environment. Our approach focuses on language-driven generality while imposing minimal assumptions. Our agents interact with environments in real-time using a generic, human-like interface.
  arXiv  Detail & Related papers  (2024-03-13T17:50:32Z)
• On Automating Video Game Regression Testing by Planning and Learning [3.746904317622708]
  We propose a method and workflow for automating regression testing of certain video game aspects. The basic idea is to use detailed game logs and incremental action model learning techniques to maintain a formal model. This paper presents the first step towards minimizing or even eliminating the need for a modeling expert in the workflow.
  arXiv  Detail & Related papers  (2024-02-16T14:28:25Z)
• An Interactive Agent Foundation Model [49.77861810045509]
  We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
  arXiv  Detail & Related papers  (2024-02-08T18:58:02Z)
• Octopus: Embodied Vision-Language Programmer from Environmental Feedback [58.04529328728999]
  Embodied vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning. To bridge this gap, we introduce Octopus, an embodied vision-language programmer that uses executable code generation as a medium to connect planning and manipulation. Octopus is designed to 1) proficiently comprehend an agent's visual and textual task objectives, 2) formulate intricate action sequences, and 3) generate executable code.
  arXiv  Detail & Related papers  (2023-10-12T17:59:58Z)
• Learning of Generalizable and Interpretable Knowledge in Grid-Based Reinforcement Learning Environments [5.217870815854702]
  We propose using program synthesis to imitate reinforcement learning policies. We adapt the state-of-the-art program synthesis system DreamCoder for learning concepts in grid-based environments.
  arXiv  Detail & Related papers  (2023-09-07T11:46:57Z)
• MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge [70.47759528596711]
  We introduce MineDojo, a new framework built on the popular Minecraft game. We propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward.
  arXiv  Detail & Related papers  (2022-06-17T15:53:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.