SAGE: Smart home Agent with Grounded Execution

• URL: http://arxiv.org/abs/2311.00772v2
• Date: Fri, 19 Jan 2024 17:14:25 GMT
• Title: SAGE: Smart home Agent with Grounded Execution
• Abstract summary: SAGE (Smart Home Agent with Grounded Execution) overcomes limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. We introduce a benchmark of 50 new and challenging smart home tasks where SAGE achieves a 75% success rate.
• Score: 1.5256015470528725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The common sense reasoning abilities and vast general knowledge of Large Language Models (LLMs) make them a natural fit for interpreting user requests in a Smart Home assistant context. LLMs, however, lack specific knowledge about the user and their home limit their potential impact. SAGE (Smart Home Agent with Grounded Execution), overcomes these and other limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. SAGE controls this process through a dynamically constructed tree of LLM prompts, which help it decide which action to take next, whether an action was successful, and when to terminate the process. The SAGE action set augments an LLM's capabilities to support some of the most critical requirements for a Smart Home assistant. These include: flexible and scalable user preference management ("is my team playing tonight?"), access to any smart device's full functionality without device-specific code via API reading "turn down the screen brightness on my dryer", persistent device state monitoring ("remind me to throw out the milk when I open the fridge"), natural device references using only a photo of the room ("turn on the light on the dresser"), and more. We introduce a benchmark of 50 new and challenging smart home tasks where SAGE achieves a 75% success rate, significantly outperforming existing LLM-enabled baselines (30% success rate).

Related papers

• GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (V1.0) [56.81743709880371]
  Long-horizon large language model (LLM) agents are fundamentally limited by context.<n>We argue that long-horizon performance is determined not by context length, but by how much decision-relevant information is maintained within a finite context budget.<n>We present GenericAgent (GA), a general-purpose, self-evolving LLM agent system built around a single principle: information density and context.
  arXiv  Detail & Related papers  (2026-04-18T17:59:15Z)
• LLM Agents Beyond Utility: An Open-Ended Perspective [50.809163251551894]
  We augment a pretrained LLM agent with the ability to generate its own tasks, accumulate knowledge, and interact extensively with its environment.<n>It can reliably follow complex multi-step instructions, store and reuse information across runs, and propose and solve its own tasks.<n>It remains sensitive to prompt design, prone to repetitive task generation, and unable to form self-representations.
  arXiv  Detail & Related papers  (2025-10-16T10:46:54Z)
• A Case for Declarative LLM-friendly Interfaces for Improved Efficiency of Computer-Use Agents [7.687846129934911]
  We propose Goal-Oriented Interface (GOI), a novel abstraction that transforms existing GUIs into three declarative primitives.<n>GOI does not require modifying the application source code or relying on application programming interfaces (APIs)
  arXiv  Detail & Related papers  (2025-10-06T09:14:58Z)
• HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices [16.797283740872835]
  Large language models (LLMs) have the potential to revolutionize smart home assistants.<n>We introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices.
  arXiv  Detail & Related papers  (2025-05-26T07:47:39Z)
• LLM Enhancer: Merged Approach using Vector Embedding for Reducing Large Language Model Hallucinations with External Knowledge [0.0]
  Large Language Models (LLMs) have demonstrated the capability to generate human like, natural responses across a range of tasks. This paper introduces the LLM ENHANCER system, designed to integrate multiple online sources such as Google, Wikipedia, and DuckDuckGo to enhance data accuracy.
  arXiv  Detail & Related papers  (2025-04-29T19:27:04Z)
• SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World [50.937342998351426]
  Chain-of-User-Thought (COUT) is a novel embodied reasoning paradigm. We introduce SmartAgent, an agent framework perceiving cyber environments and reasoning personalized requirements. Our work is the first to formulate the COUT process, serving as a preliminary attempt towards embodied personalized agent learning.
  arXiv  Detail & Related papers  (2024-12-10T12:40:35Z)
• Learning to Ask: When LLM Agents Meet Unclear Instruction [55.65312637965779]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• MEGen: Generative Backdoor in Large Language Models via Model Editing [56.46183024683885]
  Large language models (LLMs) have demonstrated remarkable capabilities. Their powerful generative abilities enable flexible responses based on various queries or instructions. This paper proposes an editing-based generative backdoor, named MEGen, aiming to create a customized backdoor for NLP tasks with the least side effects.
  arXiv  Detail & Related papers  (2024-08-20T10:44:29Z)
• CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only [21.054681757006385]
  We propose an agent that perceives its environment solely through screenshot images. By leveraging the reasoning capability of the Large Language Models, we eliminate the need for large-scale human demonstration data. Agent achieves an average success rate of 94.5% on MiniWoB++ and an average task score of 62.3 on WebShop.
  arXiv  Detail & Related papers  (2024-06-11T05:21:20Z)
• Are you still on track!? Catching LLM Task Drift with Activations [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users. We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set. We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• Human-Centered LLM-Agent User Interface: A Position Paper [8.675534401018407]
  Large Language Model (LLM) -in-the-loop applications have been shown to effectively interpret the human user's commands. A user mostly ignorant to the underlying tools/systems should be able to work with a LAUI to discover an emergent workflow.
  arXiv  Detail & Related papers  (2024-05-19T13:02:45Z)
• GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications [46.85306320942487]
  Large Language Models (LLMs) are evolving to actively engage with tools and performing actions on real-world applications and services. Today, humans verify the correctness and appropriateness of the LLM-generated outputs before putting them into real-world execution. This poses significant challenges as code comprehension is well known to be notoriously difficult. In this paper, we study how humans can efficiently collaborate with, delegate to, and supervise autonomous LLMs in the future.
  arXiv  Detail & Related papers  (2024-04-10T11:17:33Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models [14.015334285802725]
  In more natural communication, human speech is unconstrained, describing goals rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes.
  arXiv  Detail & Related papers  (2023-05-16T20:52:04Z)
• Low-code LLM: Graphical User Interface over Large Language Models [115.08718239772107]
  This paper introduces a novel human-LLM interaction framework, Low-code LLM. It incorporates six types of simple low-code visual programming interactions to achieve more controllable and stable responses. We highlight three advantages of the low-code LLM: user-friendly interaction, controllable generation, and wide applicability.
  arXiv  Detail & Related papers  (2023-04-17T09:27:40Z)
• "Get ready for a party": Exploring smarter smart spaces with help from large language models [9.62814345236243]
  We show that recent task-agnostic large language models (LLMs) embody a vast amount of cross-domain, sometimes unpredictable contextual knowledge. Our work hints at the promise of LLM-driven systems for context-awareness in smart environments.
  arXiv  Detail & Related papers  (2023-03-24T16:51:08Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.