Latent-IM: Latent Interaction Management for Speech LLMs

• URL: http://arxiv.org/abs/2607.26928v1
• Date: Wed, 29 Jul 2026 13:56:28 GMT
• Title: Latent-IM: Latent Interaction Management for Speech LLMs
• Abstract summary: We introduce Latent-IM, an internal dialogue-management framework for choosing and deploying conversational moves.<n>We use this control to reproduce human move choices, improving average end-to-end move accuracy by 12.5 points over the unsteered backbone.
• Score: 4.733799057798266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical spoken dialogue systems often separated dialogue management from response realization: a policy selected the next dialogue action, and a generation component expressed that action. As dialogue systems shift toward LLMs, this decomposition has largely disappeared into the model's hidden representations. We ask whether an LLM-internal analogue of state estimation and action control can be recovered for conversational moves such as acknowledging, checking, querying, explaining, and replying. We formulate move control as two coupled problems: selection, predicting the appropriate next move from the dialogue context, and realization, causally producing a chosen move at generation time. We introduce Latent-IM, an internal dialogue-management framework that provides a general interface for choosing and deploying conversational moves under different objectives. Here, we use this control to reproduce human move choices, improving average end-to-end move accuracy by 12.5 points over the unsteered backbone while performing comparably to fine-tuning.

Related papers

• Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager [0.7499722271664147]
  We propose a framework that integrates large language models (LLMs) with an RL-based dialogue manager for open-ended dialogue with a specific goal.<n>We employ hierarchical reinforcement learning to model the structured phases of dialogue and employ meta-learning to enhance adaptability across diverse user profiles.
  arXiv  Detail & Related papers  (2025-06-24T14:15:26Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Games [7.946510318969307]
  This study focuses on conversational norms such as adjacency pairs and turn-taking found in conversation analysis.<n>We propose a new framework called "Murder Mystery Agents" that applies these norms to AI agents' dialogue control.<n>The proposed framework integrates next speaker selection based on adjacency pairs and a self-selection mechanism.
  arXiv  Detail & Related papers  (2024-12-06T10:45:54Z)
• DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling [73.08187964426823]
  Large language models (LLMs) enabled dialogue systems have become one of the central modes in human-machine interaction.<n>This paper introduces a new research task--$textbfD$ialogue $textbfE$lement $textbfMO$deling.<n>We propose a novel benchmark, $textbfDEMO$, designed for a comprehensive dialogue modeling and assessment.
  arXiv  Detail & Related papers  (2024-12-06T10:01:38Z)
• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents [121.46051697742608]
  We introduce a new dialogue policy planning paradigm to strategize dialogue problems with a tunable language model plug-in named PPDPP. Specifically, we develop a novel training framework to facilitate supervised fine-tuning over available human-annotated data. PPDPP consistently and substantially outperforms existing approaches on three different proactive dialogue applications.
  arXiv  Detail & Related papers  (2023-11-01T03:20:16Z)
• JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy Learning [53.83063435640911]
  Dialogue policy learning (DPL) is a crucial component of dialogue modelling. We introduce a novel framework, JoTR, to generate flexible dialogue actions. Unlike traditional methods, JoTR formulates a word-level policy that allows for a more dynamic and adaptable dialogue action generation.
  arXiv  Detail & Related papers  (2023-09-01T03:19:53Z)
• Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration [72.04629217161656]
  This work focuses on three aspects of proactive dialogue systems: clarification, target-guided, and non-collaborative dialogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-of-Thought prompting scheme.
  arXiv  Detail & Related papers  (2023-05-23T02:49:35Z)
• Controllable Mixed-Initiative Dialogue Generation through Prompting [50.03458333265885]
  Mixed-initiative dialogue tasks involve repeated exchanges of information and conversational control. Agents gain control by generating responses that follow particular dialogue intents or strategies, prescribed by a policy planner. Standard approach has been fine-tuning pre-trained language models to perform generation conditioned on these intents. We instead prompt large language models as a drop-in replacement to fine-tuning on conditional generation.
  arXiv  Detail & Related papers  (2023-05-06T23:11:25Z)
• Dialog-to-Actions: Building Task-Oriented Dialogue System via Action-Level Generation [7.110201160927713]
  We propose a task-oriented dialogue system via action-level generation. Specifically, we first construct dialogue actions from large-scale dialogues and represent each natural language (NL) response as a sequence of dialogue actions. We train a Sequence-to-Sequence model which takes the dialogue history as input and outputs sequence of dialogue actions.
  arXiv  Detail & Related papers  (2023-04-03T11:09:20Z)
• Act-Aware Slot-Value Predicting in Multi-Domain Dialogue State Tracking [5.816391291790977]
  Dialogue state tracking (DST) aims to track human-machine interactions and generate state representations for managing the dialogue. Recent advances in machine reading comprehension predict both categorical and non-categorical types of slots for dialogue state tracking. We formulate and incorporate dialogue acts, and leverage recent advances in machine reading comprehension to predict both categorical and non-categorical types of slots for dialogue state tracking.
  arXiv  Detail & Related papers  (2022-08-04T05:18:30Z)
• Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management [29.57382819573169]
  We focus on devising a policy that chooses which dialogue action to respond to the user. The sequential system decision-making process can be abstracted into a partially observable Markov decision process. In the past few years, there are many deep reinforcement learning (DRL) algorithms, which use neural networks (NN) as function approximators.
  arXiv  Detail & Related papers  (2020-09-22T05:39:31Z)
• Controlling Dialogue Generation with Semantic Exemplars [55.460082747572734]
  We present an Exemplar-based Dialogue Generation model, EDGE, that uses the semantic frames present in exemplar responses to guide generation. We show that controlling dialogue generation based on the semantic frames of exemplars, rather than words in the exemplar itself, improves the coherence of generated responses.
  arXiv  Detail & Related papers  (2020-08-20T17:02:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.