SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

• URL: http://arxiv.org/abs/2606.01912v1
• Date: Mon, 01 Jun 2026 08:48:15 GMT
• Title: SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes
• Abstract summary: We introduce SMH-Bench, a comprehensive benchmark for evaluating Large Language Models (LLMs) in smart-home environments.<n>Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories.
• Score: 21.9224048962238
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferences, and multi-device interactions. However, existing smart-home benchmarks often focus on static instruction-to-API mapping or limited simulations, failing to evaluate whether LLMs can reason, interact, and act reliably in realistic household scenarios. To address these limitations, we introduce SMH-Bench, a comprehensive benchmark for evaluating LLMs in smart-home environments. Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories. It further stratifies tasks across simple, medium and complex homes, ranging from small apartments to dense multi-room environments with 135 devices. Experiments show that although frontier LLMs achieve strong performance on explicit control and query tasks, they still exhibit significant weaknesses in automation task scheduling, ambiguity handling and personalized reasoning, especially as home complexity increases. We hope SMH-Bench will facilitate the development of more reliable, context-aware, and practically deployable smart-home agents.

Related papers

• AGI Maze as a Benchmark Framework for World-Modeling Agents [0.0]
  Large language models (LLMs) are powerful pattern-completion systems, but their default operating mode does not reliably produce persistent, manipulable representations of an external world.<n>AGI Maze is a lightweight framework for building such environments without requiring high-dimensional sensory inputs.
  arXiv  Detail & Related papers  (2026-07-01T08:43:08Z)
• LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks [58.3639630490749]
  We introduce LiveClawBench, a benchmark to evaluate LLM agents on real-world assistant tasks.<n>Based on an analysis of various realClaw usage cases, we derive a Triple-Axis Complexity Framework.<n>We construct a pilot benchmark with explicit complexity-factor annotations, covering real-world assistant tasks.
  arXiv  Detail & Related papers  (2026-03-20T16:08:21Z)
• SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts [21.21657361928802]
  A critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state.<n>This involves two key requirements: the LLM must accurately determine whether an anomalous condition is present, and provide either a clear explanation or actionable suggestions.<n>To enhance the anomaly detection capabilities of next-generation LLM-based smart home assistants, we introduce SmartBench.
  arXiv  Detail & Related papers  (2026-02-24T16:07:44Z)
• SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents [10.925787133867951]
  SimuHome is a time-accelerated home environment that simulates smart devices, supports API calls, and reflects changes in environmental variables.<n>SimuHome provides a high-fidelity environment, and agents validated in SimuHome can be deployed on real Matter-compliant devices with minimal adaptation.<n>Our evaluation of 11 agents under a unified ReAct framework reveals that while models perform well on simple tasks, they struggle with latent intent inference, state verification, and especially temporal scheduling.
  arXiv  Detail & Related papers  (2025-09-29T04:54:20Z)
• HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices [16.797283740872835]
  Large language models (LLMs) have the potential to revolutionize smart home assistants.<n>We introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices.
  arXiv  Detail & Related papers  (2025-05-26T07:47:39Z)
• FamilyTool: A Multi-hop Personalized Tool Use Benchmark [93.80355496575281]
  FamilyTool is a benchmark grounded in a family-based knowledge graph (KG) that simulates personalized, multi-hop tool use scenarios.<n> Experiments reveal significant performance gaps in state-of-the-art Large Language Models (LLMs)<n>FamilyTool serves as a critical resource for evaluating and advancing LLM agents' reasoning, adaptability, and scalability in complex, dynamic environments.
  arXiv  Detail & Related papers  (2025-04-09T10:42:36Z)
• Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning [69.5875073447454]
  This paper advances motion agents empowered by large language models (LLMs) toward autonomous navigation in dynamic and cluttered environments.<n>Our training-free framework supports multi-agent coordination, closed-loop replanning, and dynamic obstacle avoidance without retraining or fine-tuning.
  arXiv  Detail & Related papers  (2025-03-10T13:39:09Z)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [63.43699771428243]
  EmbodiedBench is an extensive benchmark designed to evaluate vision-driven embodied agents.<n>We evaluated 24 leading proprietary and open-source MLLMs within EmbodiedBench.<n> MLLMs excel at high-level tasks but struggle with low-level manipulation, with the best model, GPT-4o, scoring only 28.9% on average.
  arXiv  Detail & Related papers  (2025-02-13T18:11:34Z)
• ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Execution [33.252158560173655]
  ConceptAgent is a natural language-driven robotic platform designed for task execution in unstructured environments. We present innovations designed to limit shortcomings, including 1) Predicate Grounding to prevent and recover from infeasible actions, and 2) an embodied version of LLM-guided Monte Carlo Tree Search with self reflection.
  arXiv  Detail & Related papers  (2024-10-08T15:05:40Z)
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions [72.56339136017759]
  We introduce BigCodeBench, a benchmark that challenges Large Language Models (LLMs) to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks.<n>Our evaluation shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%.<n>We propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information.
  arXiv  Detail & Related papers  (2024-06-22T15:52:04Z)
• DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent) [73.10899129264375]
  This paper explores DoraemonGPT, a comprehensive and conceptually elegant system driven by LLMs to understand dynamic scenes.<n>Given a video with a question/task, DoraemonGPT begins by converting the input video into a symbolic memory that stores task-related attributes.<n>We extensively evaluate DoraemonGPT's effectiveness on three benchmarks and several in-the-wild scenarios.
  arXiv  Detail & Related papers  (2024-01-16T14:33:09Z)
• AgentBench: Evaluating LLMs as Agents [99.12825098528212]
  Large Language Model (LLM) as agents has been widely acknowledged recently.<n>We present AgentBench, a benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
  arXiv  Detail & Related papers  (2023-08-07T16:08:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.