"Get ready for a party": Exploring smarter smart spaces with help from large language models

• URL: http://arxiv.org/abs/2303.14143v1
• Date: Fri, 24 Mar 2023 16:51:08 GMT
• Title: "Get ready for a party": Exploring smarter smart spaces with help from large language models
• Abstract summary: We show that recent task-agnostic large language models (LLMs) embody a vast amount of cross-domain, sometimes unpredictable contextual knowledge. Our work hints at the promise of LLM-driven systems for context-awareness in smart environments.
• Score: 9.62814345236243
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The right response to someone who says "get ready for a party" is deeply influenced by meaning and context. For a smart home assistant (e.g., Google Home), the ideal response might be to survey the available devices in the home and change their state to create a festive atmosphere. Current practical systems cannot service such requests since they require the ability to (1) infer meaning behind an abstract statement and (2) map that inference to a concrete course of action appropriate for the context (e.g., changing the settings of specific devices). In this paper, we leverage the observation that recent task-agnostic large language models (LLMs) like GPT-3 embody a vast amount of cross-domain, sometimes unpredictable contextual knowledge that existing rule-based home assistant systems lack, which can make them powerful tools for inferring user intent and generating appropriate context-dependent responses during smart home interactions. We first explore the feasibility of a system that places an LLM at the center of command inference and action planning, showing that LLMs have the capacity to infer intent behind vague, context-dependent commands like "get ready for a party" and respond with concrete, machine-parseable instructions that can be used to control smart devices. We furthermore demonstrate a proof-of-concept implementation that puts an LLM in control of real devices, showing its ability to infer intent and change device state appropriately with no fine-tuning or task-specific training. Our work hints at the promise of LLM-driven systems for context-awareness in smart environments, motivating future research in this area.

Related papers

• The Model Is Not the Product: A Dual-Pillar Architecture for Local-First Psychological Coaching [51.56484100374058]
  This report introduces Psych LM, an iOS application that validates the thesis that, for such applications, the surrounding architecture is paramount.<n>Psych LM runs a local, on-device language model within a purpose-built, local-first runtime designed for behavioral and life-coaching applications.
  arXiv  Detail & Related papers  (2026-05-23T05:49:11Z)
• RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs [55.44696796126619]
  Earth Observation systems are designed to support domain experts who often express their requirements through vague natural language.<n>A practical EO AI system must bridge the gap between ambiguous human queries and the appropriate multi-granularity visual analysis tasks.<n>We propose RemoteAgent, an agentic framework that strategically respects the capability intrinsic boundaries of MLLMs.
  arXiv  Detail & Related papers  (2026-04-09T03:40:46Z)
• HELP: Hierarchical Embodied Language Planner for Household Tasks [75.38606213726906]
  Embodied agents tasked with complex scenarios rely heavily on robust planning capabilities.<n>Large language models equipped with extensive linguistic knowledge can play this role.<n>We propose a Hierarchical Embodied Language Planner, called HELP, consisting of a set of LLM-based agents.
  arXiv  Detail & Related papers  (2025-12-25T15:54:08Z)
• From the Laboratory to Real-World Application: Evaluating Zero-Shot Scene Interpretation on Edge Devices for Mobile Robotics [0.0]
  This paper investigates the capabilities of state-of-the-art Visual Language Models (VLMs) for the task of Scene and Action Recognition.<n>The proposed pipeline is evaluated on a diverse dataset consisting of various real-world cityscapes, on-campus and indoor scenarios.<n>The experimental evaluation discusses the potential of these small models on edge devices, with particular emphasis on challenges, weaknesses, inherent model biases and the application of the gained information.
  arXiv  Detail & Related papers  (2025-11-04T09:58:29Z)
• Plant in Cupboard, Orange on Table, Book on Shelf. Benchmarking Practical Reasoning and Situation Modelling in a Text-Simulated Situated Environment [18.256529559741075]
  Large language models (LLMs) have risen to prominence as 'chatbots' for users to interact via natural language. We have implemented a simple text-based environment that simulates, very abstractly, a household setting. Our findings show that environmental complexity and game restrictions hamper performance.
  arXiv  Detail & Related papers  (2025-02-17T12:20:39Z)
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks [94.10497337235083]
  We are first to explore the potential of prompting speech LMs in the domain of speech processing. We reformulate speech processing tasks into speech-to-unit generation tasks. We show that the prompting method can achieve competitive performance compared to the strong fine-tuning method.
  arXiv  Detail & Related papers  (2024-08-23T13:00:10Z)
• Towards Open-World Grasping with Large Vision-Language Models [5.317624228510749]
  An open-world grasping system should be able to combine high-level contextual with low-level physical-geometric reasoning. We propose OWG, an open-world grasping pipeline that combines vision-language models with segmentation and grasp synthesis models. We conduct evaluation in cluttered indoor scene datasets to showcase OWG's robustness in grounding from open-ended language.
  arXiv  Detail & Related papers  (2024-06-26T19:42:08Z)
• PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs [140.14239499047977]
  Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. We propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT) We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities.
  arXiv  Detail & Related papers  (2024-02-12T18:33:47Z)
• DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent) [73.10899129264375]
  This paper explores DoraemonGPT, a comprehensive and conceptually elegant system driven by LLMs to understand dynamic scenes. Given a video with a question/task, DoraemonGPT begins by converting the input video into a symbolic memory that stores task-related attributes. We extensively evaluate DoraemonGPT's effectiveness on three benchmarks and several in-the-wild scenarios.
  arXiv  Detail & Related papers  (2024-01-16T14:33:09Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• SAGE: Smart home Agent with Grounded Execution [1.5256015470528725]
  SAGE (Smart Home Agent with Grounded Execution) overcomes limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. We introduce a benchmark of 50 new and challenging smart home tasks where SAGE achieves a 75% success rate.
  arXiv  Detail & Related papers  (2023-11-01T18:36:28Z)
• Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorial [35.743576799998564]
  Large language models (LLMs) have become phenomenally surging, since 2018--two decades after introducing context-aware computing. In this tutorial, we demonstrate the use of texts, prompts, and autonomous agents (AutoAgents) that enable LLMs to perform context modeling and reasoning.
  arXiv  Detail & Related papers  (2023-09-24T00:15:39Z)
• Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models [14.015334285802725]
  In more natural communication, human speech is unconstrained, describing goals rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes.
  arXiv  Detail & Related papers  (2023-05-16T20:52:04Z)
• Inner Monologue: Embodied Reasoning through Planning with Language Models [81.07216635735571]
  Large Language Models (LLMs) can be applied to domains beyond natural language processing. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios.
  arXiv  Detail & Related papers  (2022-07-12T15:20:48Z)
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [111.33545170562337]
  We investigate the possibility of grounding high-level tasks, expressed in natural language, to a chosen set of actionable steps. We find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions.
  arXiv  Detail & Related papers  (2022-01-18T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.