Thoughtful Things: Building Human-Centric Smart Devices with Small Language Models

• URL: http://arxiv.org/abs/2405.03821v1
• Date: Mon, 6 May 2024 20:04:53 GMT
• Title: Thoughtful Things: Building Human-Centric Smart Devices with Small Language Models
• Abstract summary: We propose a framework for devices that take actions and explain their behaviors in response to unconstrained user commands. Our framework requires no labeled data and can be deployed on-device, with no cloud dependency. We implement two thoughtful things (a lamp and a thermostat) and deploy them on real hardware, evaluating their practical performance.
• Score: 11.28560308392842
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Everyday devices like light bulbs and kitchen appliances are now embedded with so many features and automated behaviors that they have become complicated to actually use. While such "smart" capabilities can better support users' goals, the task of learning the "ins and outs" of different devices is daunting. Voice assistants aim to solve this problem by providing a natural language interface to devices, yet such assistants cannot understand loosely-constrained commands, they lack the ability to reason about and explain devices' behaviors to users, and they rely on connectivity to intrusive cloud infrastructure. Toward addressing these issues, we propose thoughtful things: devices that leverage lightweight, on-device language models to take actions and explain their behaviors in response to unconstrained user commands. We propose an end-to-end framework that leverages formal modeling, automated training data synthesis, and generative language models to create devices that are both capable and thoughtful in the presence of unconstrained user goals and inquiries. Our framework requires no labeled data and can be deployed on-device, with no cloud dependency. We implement two thoughtful things (a lamp and a thermostat) and deploy them on real hardware, evaluating their practical performance.

Related papers

• Agentic Language-to-Objective Synthesis for Optofluidic Assembly [15.44409126318852]
  We introduce Speak-to-Objective, a modular agentic pipeline that uses a conditioned Large Language Model (LLM) to translate spoken or written commands into differentiable objective functions.<n>We demonstrate natural-fluid-programmable, light-based microscale assembly of particle patterns in a microfluidic environment.<n>Our work points toward self-driving, AI-assisted optical manufacturing platforms in which natural language, differentiable objectives, and laser-based actuation are coupled into a reusable digital workflow.
  arXiv  Detail & Related papers  (2026-05-26T20:03:54Z)
• MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes [63.12421718928034]
  We introduce MIST (the Multimodal Interactive Speech-based Tool-calling), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices.<n>We find that there is a significant gap between open-weight and closed-weight LLMs, and that even closed-weight LLMs have substantial headroom.
  arXiv  Detail & Related papers  (2026-05-07T19:57:39Z)
• iOS as Acceleration [51.56484100374058]
  We present a proof-of-concept system demonstrating a novel approach to harness an iOS device via distributed pipeline parallelism.<n>The findings of this paper highlight the potential for the improving commonplace mobile devices to provide greater contributions to machine learning.
  arXiv  Detail & Related papers  (2025-12-19T13:30:44Z)
• Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models [50.19518681574399]
  A central challenge in real-world assistive teleoperation is for the robot to infer a wide range of human intentions from user control inputs.<n>We introduce Casper, an assistive teleoperation system that leverages commonsense knowledge embedded in pre-trained visual language models.<n>We show that Casper improves task performance, reduces human cognitive load, and achieves higher user satisfaction than direct teleoperation and assistive teleoperation baselines.
  arXiv  Detail & Related papers  (2025-06-17T17:06:43Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use. MeCo captures high-level cognitive signals in the representation space, guiding when to invoke tools. Our experiments show that MeCo accurately detects LLMs' internal cognitive signals and significantly improves tool-use decision-making.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• Asynchronous Tool Usage for Real-Time Agents [61.3041983544042]
  We introduce asynchronous AI agents capable of parallel processing and real-time tool-use. Our key contribution is an event-driven finite-state machine architecture for agent execution and prompting. This work presents both a conceptual framework and practical tools for creating AI agents capable of fluid, multitasking interactions.
  arXiv  Detail & Related papers  (2024-10-28T23:57:19Z)
• Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Survey [52.00674453604779]
  This survey organizes the literature on the design and optimization of emerging structures around post-trained LMs.<n>We refer to this overarching structure as scaffolded LMs and focus on LMs that are integrated into multi-step processes with tools.
  arXiv  Detail & Related papers  (2024-10-21T18:06:25Z)
• Video as the New Language for Real-World Decision Making [100.68643056416394]
  Video data captures important information about the physical world that is difficult to express in language. Video can serve as a unified interface that can absorb internet knowledge and represent diverse tasks. We identify major impact opportunities in domains such as robotics, self-driving, and science.
  arXiv  Detail & Related papers  (2024-02-27T02:05:29Z)
• Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Models [10.892991111926573]
  We present a paradigm for human-computer interaction that revolutionizes the traditional notion of an operating system. Within this innovative framework, user requests issued to the machine are handled by an interconnected ecosystem of generative AI models. This visionary concept raises significant challenges, including privacy, security, trustability, and the ethical use of generative models.
  arXiv  Detail & Related papers  (2023-10-07T17:16:34Z)
• Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models [14.015334285802725]
  In more natural communication, human speech is unconstrained, describing goals rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes.
  arXiv  Detail & Related papers  (2023-05-16T20:52:04Z)
• Plug-and-Play Multilingual Few-shot Spoken Words Recognition [3.591566487849146]
  We propose PLiX, a multilingual and plug-and-play keyword spotting system. Our few-shot deep models are learned with millions of one-second audio clips across 20 languages. We show that PLiX can generalize to novel spoken words given as few as just one support example.
  arXiv  Detail & Related papers  (2023-05-03T18:58:14Z)
• "Get ready for a party": Exploring smarter smart spaces with help from large language models [9.62814345236243]
  We show that recent task-agnostic large language models (LLMs) embody a vast amount of cross-domain, sometimes unpredictable contextual knowledge. Our work hints at the promise of LLM-driven systems for context-awareness in smart environments.
  arXiv  Detail & Related papers  (2023-03-24T16:51:08Z)
• Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents [111.15288256221764]
  Grounded-decoding project aims to solve complex, long-horizon tasks in a robotic setting by leveraging the knowledge of both models. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon tasks in a robotic setting by leveraging the knowledge of both models.
  arXiv  Detail & Related papers  (2023-03-01T22:58:50Z)
• Toolformer: Language Models Can Teach Themselves to Use Tools [62.04867424598204]
  Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. We show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction.
  arXiv  Detail & Related papers  (2023-02-09T16:49:57Z)
• MetaNetwork: A Task-agnostic Network Parameters Generation Framework for Improving Device Model Generalization [65.02542875281233]
  We propose a novel task-agnostic framework, named MetaNetwork, for generating adaptive device model parameters from cloud without on-device training. The MetaGenerator is designed to learn a mapping function from samples to model parameters, and it can generate and deliver the adaptive parameters to the device based on samples uploaded from the device to the cloud. The MetaStabilizer aims to reduce the oscillation of the MetaGenerator, accelerate the convergence and improve the model performance during both training and inference.
  arXiv  Detail & Related papers  (2022-09-12T13:26:26Z)
• Natural Language Sentence Generation from API Specifications [5.192671914929481]
  We propose a system to generate sentences to train intent recognition models. The human-in-the-loop interaction will provide further improvement on the system.
  arXiv  Detail & Related papers  (2022-06-01T15:50:14Z)
• GenNI: Human-AI Collaboration for Data-Backed Text Generation [102.08127062293111]
  Table2Text systems generate textual output based on structured data utilizing machine learning. GenNI (Generation Negotiation Interface) is an interactive visual system for high-level human-AI collaboration in producing descriptive text.
  arXiv  Detail & Related papers  (2021-10-19T18:07:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.