MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes

• URL: http://arxiv.org/abs/2605.06897v1
• Date: Thu, 07 May 2026 19:57:39 GMT
• Title: MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes
• Abstract summary: We introduce MIST (the Multimodal Interactive Speech-based Tool-calling), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices.<n>We find that there is a significant gap between open-weight and closed-weight LLMs, and that even closed-weight LLMs have substantial headroom.
• Score: 63.12421718928034
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rise of Internet of Things (IoT) devices in the physical world necessitates voice-based interfaces capable of handling complex user experiences. While modern Large Language Models (LLMs) already demonstrate strong tool-usage capabilities, modeling real-world IoT devices presents a difficult, understudied challenge which combines modeling spatiotemporal constraints with speech inputs, dynamic state tracking, and mixed-initiative interaction patterns. We introduce MIST (the Multimodal Interactive Speech-based Tool-calling Dataset), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices. We find that there is a significant gap between open- and closed-weight multimodal LLMs on MIST, and that even frontier closed-weight LLMs have substantial headroom. We release MIST and an extensible data generation framework to build related datasets in order to facilitate research on mixed-initiative voice assistants which reason about physical world constraints.

Related papers

• A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot [0.0]
  We present an open-source Android framework for the Pepper robot.<n>We integrate end-to-end Speech-to-Speech (S2S) models to achieve low-latency interaction.<n>We implement extensive Function capabilities that elevate the Large Language Models to an agentic planner.
  arXiv  Detail & Related papers  (2026-01-09T17:33:38Z)
• MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation [59.23161833385837]
  We propose MAViD, a novel Multimodal framework for Audio-Visual Dialogue understanding and generation.<n>Our framework can generate vivid and contextually coherent long-duration dialogue interactions and accurately interpret users' multimodal queries.
  arXiv  Detail & Related papers  (2025-12-02T18:55:53Z)
• Sensing and Understanding the World over Air: A Large Multimodal Model for Mobile Networks [59.23869884913339]
  Wireless-native multi-modal large models (WMLMs) can sense and understand the physical world through multi-modal data.<n>We constructed a GPT-style WMLM model and trained it on a real-world large-scale dataset, leveraging wireless signals as an anchor modality for contrastive learning.
  arXiv  Detail & Related papers  (2025-11-17T07:33:46Z)
• InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue [35.99134148462425]
  We introduce Interactive Omni, a unified and open-source omni-modal large language model for audio-visual multi-turn interaction.<n>To achieve this, we integrate the vision encoder, audio encoder, large language model, and speech decoder into a unified model for understanding and generation tasks.<n>We design a multi-stage training strategy to ensure robust cross-modal capabilities, including pre-training for omni-modal understanding.
  arXiv  Detail & Related papers  (2025-10-15T16:52:48Z)
• Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition [57.131546757903834]
  Lyra is an efficient MLLM that enhances multimodal abilities, including advanced long-speech comprehension, sound understanding, cross-modality efficiency, and seamless speech interaction.<n>Lyra achieves state-of-the-art performance on various vision-language, vision-speech, and speech-language benchmarks, while also using fewer computational resources and less training data.
  arXiv  Detail & Related papers  (2024-12-12T17:50:39Z)
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training [55.54020926284334]
  Multimodal Large Language Models (MLLMs) have recently received substantial interest, which shows their emerging potential as general-purpose models for various vision-language tasks. Retrieval augmentation techniques have proven to be effective plugins for both LLMs and MLLMs. In this study, we propose multimodal adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training (RA-BLIP), a novel retrieval-augmented framework for various MLLMs.
  arXiv  Detail & Related papers  (2024-10-18T03:45:19Z)
• Language Model Can Listen While Speaking [17.584201137311286]
  Listen-while-speaking language model (LSLM) is an end-to-end system equipped with both listening and speaking channels. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems.
  arXiv  Detail & Related papers  (2024-08-05T16:47:22Z)
• A General-Purpose Device for Interaction with LLMs [3.052172365469752]
  This paper investigates integrating large language models (LLMs) with advanced hardware. We focus on developing a general-purpose device designed for enhanced interaction with LLMs.
  arXiv  Detail & Related papers  (2024-08-02T23:43:29Z)
• LLMs Meet Multimodal Generation and Editing: A Survey [89.76691959033323]
  This survey elaborates on multimodal generation and editing across various domains, comprising image, video, 3D, and audio. We summarize the notable advancements with milestone works in these fields and categorize these studies into LLM-based and CLIP/T5-based methods. We dig into tool-augmented multimodal agents that can leverage existing generative models for human-computer interaction.
  arXiv  Detail & Related papers  (2024-05-29T17:59:20Z)
• LEGENT: Open Platform for Embodied Agents [60.71847900126832]
  We introduce LEGENT, an open, scalable platform for developing embodied agents using Large Language Models (LLMs) and Large Multimodal Models (LMMs) LEGENT offers a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface. In experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks.
  arXiv  Detail & Related papers  (2024-04-28T16:50:12Z)
• Chat with the Environment: Interactive Multimodal Perception Using Large Language Models [19.623070762485494]
  Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment.
  arXiv  Detail & Related papers  (2023-03-14T23:01:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.