Automation Configuration in Smart Home Systems: Challenges and Opportunities

• URL: http://arxiv.org/abs/2408.04755v1
• Date: Thu, 8 Aug 2024 21:05:44 GMT
• Title: Automation Configuration in Smart Home Systems: Challenges and Opportunities
• Abstract summary: Home Assistant (HA) is one of the most popular platforms for smart homes. It allows end-users to smartify homes by (S1) integrating selected devices into the system, and (S2) creating YAML files to control those devices. Unfortunately, due to the diversity of devices and complexity of automatic configurations, many users have difficulty correctly creating YAML files.
• Score: 9.716646750290913
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As the innovation of smart devices and internet-of-things (IoT), smart homes have become prevalent. People tend to transform residences into smart homes by customizing off-the-shelf smart home platforms, instead of creating IoT systems from scratch. Among the alternatives, Home Assistant (HA) is one of the most popular platforms. It allows end-users (i.e., home residents) to smartify homes by (S1) integrating selected devices into the system, and (S2) creating YAML files to control those devices. Unfortunately, due to the diversity of devices and complexity of automatic configurations, many users have difficulty correctly creating YAML files. Consequently, their smart homes may not work as expected, causing frustration and concern in users. This paper presents a novel study on issues of YAML-based automation configuration in smart homes (issues related to S2). We mined the online forum Home Assistant Community for discussion threads related to automation configuration. By manually inspecting 190 threads, we revealed 3 categories of concerns: implementation, optimization, and debugging. Under each category, we classified discussions based on the issue locations and technical concepts involved. Among debugging discussions, we further classified discussions based on users' resolution strategies; we also applied existing analysis tools to buggy YAML files, to assess the tool effectiveness. Our study reveals the common challenges faced by users and frequently applied resolution strategies. There are 129 (68%) examined issues concerning debugging, but existing tools can detect at most 14 issues and fix none. It implies that existing tools provide limited assistance in automation configuration. Our research sheds light on future directions in smart home development.

Related papers

• SmartHomeSecure: Automated Detection and Repair of Smart Home Configuration Errors Using Large Language Models [12.096717905785779]
  This paper presents SmartHomeSecure, a prototype for automated detection and repair of configuration errors.<n>The system is implemented as a modular web application with four layers: UI Shell, Feature Orchestrator, Domain Engine, and Integration Layer.<n>Three models achieved 100% error detection accuracy, with success rates ranging from 87% to 93%.
  arXiv  Detail & Related papers  (2026-07-07T19:29:10Z)
• SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes [21.9224048962238]
  We introduce SMH-Bench, a comprehensive benchmark for evaluating Large Language Models (LLMs) in smart-home environments.<n>Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories.
  arXiv  Detail & Related papers  (2026-06-01T08:48:15Z)
• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing [57.7131457251794]
  We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
  arXiv  Detail & Related papers  (2026-03-26T09:27:29Z)
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• Identifying and Addressing User-level Security Concerns in Smart Homes Using "Smaller" LLMs [0.9868505042185401]
  We aim to identify and address the major user-level security concerns in smart homes.<n>Specifically, we develop a novel dataset of Q&A from public forums.<n>We fine-tune relatively "smaller" transformer models, such as T5 and Flan-T5, on this dataset to build a QA system tailored for smart home security.
  arXiv  Detail & Related papers  (2025-09-23T18:47:59Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices [16.797283740872835]
  Large language models (LLMs) have the potential to revolutionize smart home assistants.<n>We introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices.
  arXiv  Detail & Related papers  (2025-05-26T07:47:39Z)
• FamilyTool: A Multi-hop Personalized Tool Use Benchmark [94.1158032740113]
  We introduce FamilyTool, a novel benchmark grounded in a family-based knowledge graph (KG) FamilyTool challenges Large Language Models with queries spanning 1 to 3 relational hops. Experiments reveal significant performance gaps in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-04-09T10:42:36Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use. MeCo captures high-level cognitive signals in the representation space, guiding when to invoke tools. Our experiments show that MeCo accurately detects LLMs' internal cognitive signals and significantly improves tool-use decision-making.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• AutoIoT: Automated IoT Platform Using Large Language Models [14.481067833984474]
  AutoIoT is an automated IoT platform based on Large Language Models (LLMs) and formal verification techniques. We show how AutoIoT can help users generate conflict-free automation rules and assist developers in generating codes for conflict detection.
  arXiv  Detail & Related papers  (2024-11-16T02:02:01Z)
• SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation [89.24729958546168]
  We present SPA-Bench, a comprehensive SmartPhone Agent Benchmark designed to evaluate (M)LLM-based agents. SPA-Bench offers three key contributions: A diverse set of tasks covering system and third-party apps in both English and Chinese, focusing on features commonly used in daily routines. A novel evaluation pipeline that automatically assesses agent performance across multiple dimensions, encompassing seven metrics related to task completion and resource consumption.
  arXiv  Detail & Related papers  (2024-10-19T17:28:48Z)
• SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? [64.34184587727334]
  We propose SWE-bench Multimodal to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization.
  arXiv  Detail & Related papers  (2024-10-04T18:48:58Z)
• MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants [64.41695570145673]
  We propose MemSim, a Bayesian simulator designed to automatically construct reliable questions and answers (QAs) from generated user messages. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-30T10:19:04Z)
• Multi-LLM QA with Embodied Exploration [55.581423861790945]
  We investigate the use of Multi-Embodied LLM Explorers (MELE) for question-answering in an unknown environment. Multiple LLM-based agents independently explore and then answer queries about a household environment. We analyze different aggregation methods to generate a single, final answer for each query.
  arXiv  Detail & Related papers  (2024-06-16T12:46:40Z)
• AutoSurvey: Large Language Models Can Automatically Write Surveys [77.0458309675818]
  This paper introduces AutoSurvey, a speedy and well-organized methodology for automating the creation of comprehensive literature surveys. Traditional survey paper creation faces challenges due to the vast volume and complexity of information. Our contributions include a comprehensive solution to the survey problem, a reliable evaluation method, and experimental validation demonstrating AutoSurvey's effectiveness.
  arXiv  Detail & Related papers  (2024-06-10T12:56:06Z)
• DREAM: Debugging and Repairing AutoML Pipelines [39.83914420717843]
  We design and implement DREAM, an automatic debug and repairing system for AutoML systems. It monitors the process of AutoML to collect detailed feedback and automatically repairs bugs by expanding search space. Our evaluation results show that DREAM can effectively and efficiently repair AutoML bugs.
  arXiv  Detail & Related papers  (2023-12-31T02:45:17Z)
• AgentSims: An Open-Source Sandbox for Large Language Model Evaluation [9.156652770482268]
  Existing evaluation methods suffer from following shortcomings: (1) constrained evaluation abilities, (2) vulnerable benchmarks, (3) unobjective metrics. We suggest that task-based evaluation, where LLM agents complete tasks in a simulated environment, is a one-for-all solution to solve above problems. AgentSims is an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in.
  arXiv  Detail & Related papers  (2023-08-08T03:59:28Z)
• HomeRobot: Open-Vocabulary Mobile Manipulation [107.05702777141178]
  Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch.
  arXiv  Detail & Related papers  (2023-06-20T14:30:32Z)
• Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models [14.015334285802725]
  In more natural communication, human speech is unconstrained, describing goals rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes.
  arXiv  Detail & Related papers  (2023-05-16T20:52:04Z)
• SafeAccess+: An Intelligent System to make Smart Home Safer and Americans with Disability Act Compliant [0.0]
  Smart homes equipped with ADA compliant appliances and services are critical for people with disabilities. People with disabilities often feel insecure to respond when someone knocks on the door or rings the doorbell. We present an intelligent system called "SafeAccess+" to build safer and ADA compliant premises.
  arXiv  Detail & Related papers  (2021-09-14T22:39:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.