ProactiveLLM: Learning Active Interaction for Streaming Large Language Models

• URL: http://arxiv.org/abs/2606.00523v1
• Date: Sat, 30 May 2026 04:31:29 GMT
• Title: ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
• Abstract summary: We propose ProactiveLLM, which achieves active interaction by leveraging the model's endogenous states to guide interaction decisions.<n>ProactiveLLM significantly reduces interaction latency while maintaining quality, validating its capacity for dynamic and active interaction.
• Score: 19.8873622850537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard Large Language Models (LLMs) follow a read-then-generate paradigm, causing unnecessary latency and computation. Streaming LLMs alleviate this issue by generating while receiving inputs, but still struggle to decide when to interact with the stream. Existing methods either hard-code interaction timing or rely on costly external alignment signals, such as timing labels, reasoning trajectories, or stronger teachers. In this paper, we propose ProactiveLLM, which achieves active interaction by leveraging the model's endogenous states to guide interaction decisions. The model first learns to perceive semantic sufficiency from partial inputs through two complementary training mechanisms: mask-based streaming modeling and synchronized privileged self-distillation (SPSD). The former applies monotonic random masking to the input during training, simulating progressively revealed streaming inputs and enabling the model to learn local semantic dependencies from partial-input views. The latter aligns the partial-context student view with a full-context teacher view generated by the same evolving model, allowing privileged full-context evidence to guide the student's understanding under incomplete observations. Together, these mechanisms induce endogenous sufficiency cues without requiring external teachers or annotations, providing a versatile foundation for the plug-and-play integration of diverse decision heads. Extensive evaluation across text and speech streaming tasks confirms that ProactiveLLM significantly reduces interaction latency while maintaining quality, validating its capacity for dynamic and active interaction. Code is publicly available at https://github.com/EIT-NLP/StreamingLLM/tree/main/ProactiveLLM.

Related papers

• StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models [84.17128030384099]
  StreamPI is a streaming multimodal temporal modeling framework.<n>It equips single-frame VLA with temporal reasoning capability without introducing any additional parameters.<n> Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
  arXiv  Detail & Related papers  (2026-08-26T17:33:19Z)
• A Model Merging Approach for Continual MLLM Unlearning [22.83414107472909]
  Merging for Continual Unlearning (MCU) is an approach that dynamically merges multiple one-shot unlearning adapters into a unified adapter upon receiving each new unlearning request.<n>MCU achieves superior unlearning effectiveness while preserving both retained knowledge and general multimodal utility.
  arXiv  Detail & Related papers  (2026-08-05T07:37:23Z)
• Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering [87.43777061308658]
  We analyze predictive behavior in full spoken language models (FDSLMs)<n>We show that FDSLMs dynamically modulate their internal focus between two states: a generative state aligned with model output generation and a perceptive state aligned incoming user input.<n>During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it miss the beginning of incoming input.
  arXiv  Detail & Related papers  (2026-06-09T19:08:07Z)
• Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning [25.860185076316508]
  Multimodal Large Language Models (MLLMs) have achieved remarkable progress on vision-language tasks.<n>They may also memorize and expose sensitive or restricted knowledge, raising concerns about privacy and broader safety risks.<n>We propose Visual-Noise Guided In-Context Distillation (VGID) as a distillation-based framework for MLLM unlearning.
  arXiv  Detail & Related papers  (2026-05-26T12:49:34Z)
• Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos [70.78278988781882]
  We present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs.<n>Our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating.
  arXiv  Detail & Related papers  (2026-05-25T18:02:20Z)
• MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction [76.4461698685681]
  Mini-o 4.5 is our latest effort towards human-level realtime streaming interaction.<n>It is based on Omni-CPM, a unified streaming framework that aligns omni-modal perception and outputs along a shared temporal axis.<n>With a total of 9B parameters, Mini-o 4.5 approaches Gemini 2.5 Flash in vision computation capabilities, delivering stateofthe-art open computation performance scale.
  arXiv  Detail & Related papers  (2026-04-30T04:05:43Z)
• Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis [34.5994686982342]
  Strong semantic representations improve the convergence and generation quality of diffusion and flow models.<n>Existing approaches largely rely on external models, which require separate training, operate on misaligned objectives, and exhibit unexpected scaling behavior.<n>We introduce Self-Flow: a self-supervised flow matching paradigm that integrates representation learning within the generative framework.
  arXiv  Detail & Related papers  (2026-03-06T17:41:49Z)
• MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering [36.80441487363007]
  MLLMEraser is an input-aware, training-free framework for test-time unlearning.<n>We construct a multimodal erasure direction by contrasting adversarially perturbed, knowledge-recall image-text pairs.<n>Experiments on LLaVA-1.5 and Qwen-2.5-VL demonstrate that MLLMEraser consistently outperforms state-of-the-art MLLM unlearning baselines.
  arXiv  Detail & Related papers  (2025-10-05T14:20:17Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• Efficient Transfer Learning for Video-language Foundation Models [13.166348605993292]
  We propose a parameter-efficient Multi-modalpatio Ssupervised-Temporal Adapter (MSTA) to enhance alignment between textual and visual representations.<n>We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-Temporal learning.
  arXiv  Detail & Related papers  (2024-11-18T01:25:58Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.