HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices

• URL: http://arxiv.org/abs/2505.19628v2
• Date: Tue, 27 May 2025 07:26:36 GMT
• Title: HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices
• Abstract summary: Large language models (LLMs) have the potential to revolutionize smart home assistants.<n>We introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices.
• Score: 16.797283740872835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have the potential to revolutionize smart home assistants by enhancing their ability to accurately understand user needs and respond appropriately, which is extremely beneficial for building a smarter home environment. While recent studies have explored integrating LLMs into smart home systems, they primarily focus on handling straightforward, valid single-device operation instructions. However, real-world scenarios are far more complex and often involve users issuing invalid instructions or controlling multiple devices simultaneously. These have two main challenges: LLMs must accurately identify and rectify errors in user instructions and execute multiple user instructions perfectly. To address these challenges and advance the development of LLM-based smart home assistants, we introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices in this paper. We have experimental results on 13 distinct LLMs; e.g., GPT-4o achieves only a 0.0% success rate in the scenario of invalid multi-device instructions, revealing that the existing state-of-the-art LLMs still cannot perform well in this situation even with the help of in-context learning, retrieval-augmented generation, and fine-tuning. Our code and dataset are publicly available at https://github.com/BITHLP/HomeBench.

Related papers

• SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes [21.9224048962238]
  We introduce SMH-Bench, a comprehensive benchmark for evaluating Large Language Models (LLMs) in smart-home environments.<n>Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories.
  arXiv  Detail & Related papers  (2026-06-01T08:48:15Z)
• SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts [21.21657361928802]
  A critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state.<n>This involves two key requirements: the LLM must accurately determine whether an anomalous condition is present, and provide either a clear explanation or actionable suggestions.<n>To enhance the anomaly detection capabilities of next-generation LLM-based smart home assistants, we introduce SmartBench.
  arXiv  Detail & Related papers  (2026-02-24T16:07:44Z)
• Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning [34.70213312250216]
  We propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale language model to collaborate with large-scale language models.<n>A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy.<n>Prompt-R1 significantly outperforms baseline models across tasks.
  arXiv  Detail & Related papers  (2025-11-02T17:11:03Z)
• LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests [7.6818904666624395]
  This paper proposes a dual-LLM system and experiments with the usage of LLMs for the generation of compiler tests.<n>It is evident that LLMs possess the promising potential to generate quality compiler tests and verify them automatically.
  arXiv  Detail & Related papers  (2025-07-29T02:34:28Z)
• LLMs Get Lost In Multi-Turn Conversation [44.26588510453331]
  Large Language Models (LLMs) are conversational interfaces.<n>LLMs have the potential to assist their users not only when they can fully specify the task at hand, but also to help them define, explore, and refine what they need through multi-turn conversational exchange.
  arXiv  Detail & Related papers  (2025-05-09T15:21:44Z)
• SpecTool: A Benchmark for Characterizing Errors in Tool-Use LLMs [77.79172008184415]
  SpecTool is a new benchmark to identify error patterns in LLM output on tool-use tasks. We show that even the most prominent LLMs exhibit these error patterns in their outputs. Researchers can use the analysis and insights from SPECTOOL to guide their error mitigation strategies.
  arXiv  Detail & Related papers  (2024-11-20T18:56:22Z)
• Learning to Ask: When LLM Agents Meet Unclear Instruction [55.65312637965779]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone.<n>We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench.<n>We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error [54.954211216847135]
  Existing large language models (LLMs) only reach a correctness rate in the range of 30% to 60%. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE) STE orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory.
  arXiv  Detail & Related papers  (2024-03-07T18:50:51Z)
• Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs [60.40396361115776]
  This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in large language models (LLMs) with a slim proxy model. We employ a proxy model which has far fewer parameters, and take its answers as answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM.
  arXiv  Detail & Related papers  (2024-02-19T11:11:08Z)
• Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking [5.755004576310333]
  Large Language Model (LLM) assistants have emerged as potential alternatives to search methods for helping users navigate software. LLM assistants use vast training data from domain-specific texts, software manuals, and code repositories to mimic human-like interactions.
  arXiv  Detail & Related papers  (2024-02-12T19:49:58Z)
• MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning [40.32823306537386]
  We propose MLLM-Tool, a system incorporating open-source large language models and multi-modal encoders.<n>Our dataset features multi-modal input tools from HuggingFace.<n>Experiments reveal that our MLLM-Tool is capable of recommending appropriate tools for multi-modal instructions.
  arXiv  Detail & Related papers  (2024-01-19T14:44:37Z)
• Small LLMs Are Weak Tool Learners: A Multi-LLM Agent [73.54562551341454]
  Large Language Model (LLM) agents significantly extend the capabilities of standalone LLMs. We propose a novel approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer. This modular framework facilitates individual updates and the potential use of smaller LLMs for building each capability.
  arXiv  Detail & Related papers  (2024-01-14T16:17:07Z)
• SAGE: Smart home Agent with Grounded Execution [1.5256015470528725]
  SAGE (Smart Home Agent with Grounded Execution) overcomes limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. We introduce a benchmark of 50 new and challenging smart home tasks where SAGE achieves a 75% success rate.
  arXiv  Detail & Related papers  (2023-11-01T18:36:28Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.