Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models

• URL: http://arxiv.org/abs/2305.09802v3
• Date: Thu, 25 Jan 2024 20:04:50 GMT
• Title: Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models
• Abstract summary: In more natural communication, human speech is unconstrained, describing goals rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes.
• Score: 14.015334285802725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Smart home assistants function best when user commands are direct and well-specified (e.g., "turn on the kitchen light"), or when a hard-coded routine specifies the response. In more natural communication, however, human speech is unconstrained, often describing goals (e.g., "make it cozy in here" or "help me save energy") rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes. We empirically study the baseline quality and failure modes of LLM-created action plans with a survey of age-diverse users. We find that LLMs can reason creatively to achieve challenging goals, but they experience patterns of failure that diminish their usefulness. We address these gaps with Sasha, a smarter smart home assistant. Sasha responds to loosely-constrained commands like "make it cozy" or "help me sleep better" by executing plans to achieve user goals, e.g., setting a mood with available devices, or devising automation routines. We implement and evaluate Sasha in a hands-on user study, showing the capabilities and limitations of LLM-driven smart homes when faced with unconstrained user-generated scenarios.

Related papers

• LLM Agents Beyond Utility: An Open-Ended Perspective [50.809163251551894]
  We augment a pretrained LLM agent with the ability to generate its own tasks, accumulate knowledge, and interact extensively with its environment.<n>It can reliably follow complex multi-step instructions, store and reuse information across runs, and propose and solve its own tasks.<n>It remains sensitive to prompt design, prone to repetitive task generation, and unable to form self-representations.
  arXiv  Detail & Related papers  (2025-10-16T10:46:54Z)
• HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices [16.797283740872835]
  Large language models (LLMs) have the potential to revolutionize smart home assistants.<n>We introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices.
  arXiv  Detail & Related papers  (2025-05-26T07:47:39Z)
• Plant in Cupboard, Orange on Table, Book on Shelf. Benchmarking Practical Reasoning and Situation Modelling in a Text-Simulated Situated Environment [18.256529559741075]
  Large language models (LLMs) have risen to prominence as 'chatbots' for users to interact via natural language. We have implemented a simple text-based environment that simulates, very abstractly, a household setting. Our findings show that environmental complexity and game restrictions hamper performance.
  arXiv  Detail & Related papers  (2025-02-17T12:20:39Z)
• Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning [61.17099595835263]
  Gensors is a system that empowers users to define customized sensors supported by the reasoning capabilities of MLLMs. In a user study, participants reported significantly greater sense of control, understanding, and ease of communication when defining sensors using Gensors.
  arXiv  Detail & Related papers  (2025-01-27T01:47:57Z)
• SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World [50.937342998351426]
  Chain-of-User-Thought (COUT) is a novel embodied reasoning paradigm. We introduce SmartAgent, an agent framework perceiving cyber environments and reasoning personalized requirements. Our work is the first to formulate the COUT process, serving as a preliminary attempt towards embodied personalized agent learning.
  arXiv  Detail & Related papers  (2024-12-10T12:40:35Z)
• Learning to Ask: When LLMs Meet Unclear Instruction [49.256630152684764]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• Automation Configuration in Smart Home Systems: Challenges and Opportunities [9.716646750290913]
  Home Assistant (HA) is one of the most popular platforms for smart homes. It allows end-users to smartify homes by (S1) integrating selected devices into the system, and (S2) creating YAML files to control those devices. Unfortunately, due to the diversity of devices and complexity of automatic configurations, many users have difficulty correctly creating YAML files.
  arXiv  Detail & Related papers  (2024-08-08T21:05:44Z)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs [38.86647602211699]
  AI assistants such as ChatGPT are trained to respond to users by saying, "I am a large language model" Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness.
  arXiv  Detail & Related papers  (2024-07-05T17:57:02Z)
• Thoughtful Things: Building Human-Centric Smart Devices with Small Language Models [11.28560308392842]
  We propose a framework for devices that take actions and explain their behaviors in response to unconstrained user commands. Our framework requires no labeled data and can be deployed on-device, with no cloud dependency. We implement two thoughtful things (a lamp and a thermostat) and deploy them on real hardware, evaluating their practical performance.
  arXiv  Detail & Related papers  (2024-05-06T20:04:53Z)
• DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent) [73.10899129264375]
  This paper explores DoraemonGPT, a comprehensive and conceptually elegant system driven by LLMs to understand dynamic scenes. Given a video with a question/task, DoraemonGPT begins by converting the input video into a symbolic memory that stores task-related attributes. We extensively evaluate DoraemonGPT's effectiveness on three benchmarks and several in-the-wild scenarios.
  arXiv  Detail & Related papers  (2024-01-16T14:33:09Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• SAGE: Smart home Agent with Grounded Execution [1.5256015470528725]
  SAGE (Smart Home Agent with Grounded Execution) overcomes limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. We introduce a benchmark of 50 new and challenging smart home tasks where SAGE achieves a 75% success rate.
  arXiv  Detail & Related papers  (2023-11-01T18:36:28Z)
• "Get ready for a party": Exploring smarter smart spaces with help from large language models [9.62814345236243]
  We show that recent task-agnostic large language models (LLMs) embody a vast amount of cross-domain, sometimes unpredictable contextual knowledge. Our work hints at the promise of LLM-driven systems for context-awareness in smart environments.
  arXiv  Detail & Related papers  (2023-03-24T16:51:08Z)
• Guiding Pretraining in Reinforcement Learning with Large Language Models [133.32146904055233]
  We describe a method that uses background knowledge from text corpora to shape exploration. This method, called ELLM, rewards an agent for achieving goals suggested by a language model. By leveraging large-scale language model pretraining, ELLM guides agents toward human-meaningful and plausibly useful behaviors without requiring a human in the loop.
  arXiv  Detail & Related papers  (2023-02-13T21:16:03Z)
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [111.33545170562337]
  We investigate the possibility of grounding high-level tasks, expressed in natural language, to a chosen set of actionable steps. We find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions.
  arXiv  Detail & Related papers  (2022-01-18T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.