IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams

• URL: http://arxiv.org/abs/2605.27074v1
• Date: Tue, 26 May 2026 14:23:25 GMT
• Title: IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams
• Abstract summary: multimodal large language models (MLLMs) achieve strong performance on reactive question answering, but real-world streaming assistants require proactive reasoning over continuous visual inputs.<n>We introduce IPIBench, the first benchmark for evaluating Interactive Intelligence of MLLMs under streaming video settings.<n>We propose IPI-Agent, a training-free agentic framework with an interaction-control policy and a temporal-gating mechanism for stabilizing proactive triggering and coordinating multi-turn interactions.
• Score: 13.326802573561748
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent multimodal large language models (MLLMs) achieve strong performance on reactive question answering, but real-world streaming assistants require proactive reasoning over continuous visual inputs. Existing benchmarks mainly study reactive or proactive interactions in isolated single-turn settings, overlooking dynamic multi-turn scenarios where users may add, modify, or cancel proactive requests alongside interleaved reactive queries. To address this gap, we introduce IPIBench, the first benchmark for evaluating Interactive Proactive Intelligence of MLLMs under streaming video settings. IPIBench covers proactive monitoring, proactive task management, and interleaved reactive-proactive requests. Evaluations on representative MLLMs reveal two major limitations: unstable proactive triggering and weak coordination between reactive and proactive behaviors. We further propose IPI-Agent, a training-free agentic framework with an interaction-control policy and a temporal-gating mechanism for stabilizing proactive triggering and coordinating multi-turn interactions. Experiments show that IPI-Agent consistently improves existing MLLMs across all benchmark settings.

Related papers

• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• Learning to Watch: Active Video Anomaly Understanding via Interleaved Policy Optimization [56.150184676745205]
  Video anomaly understanding (VAU) relies on sparse, context-dependent cues.<n>We propose $Anomtext-$, a closed-loop framework that reconceptualizes video understanding as an active sequential decision-making process.<n>Inspired by human video-reviewing behavior, this framework unifies internal cognitive reasoning and strategic evidence acquisition into an interleaved policy.
  arXiv  Detail & Related papers  (2026-07-01T08:41:49Z)
• OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents [74.20327254615854]
  Large language model agents interleave reasoning, action selection, and observation to solve sequential decision-making tasks.<n>Existing inference-time adaptation methods for LLM agents mainly rely on prompting or retrieval.<n>We propose OLIVIA, an inference-time action adaptation framework for ReAct-style agents.
  arXiv  Detail & Related papers  (2026-05-11T19:28:20Z)
• A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agents [18.524016493711386]
  Large Language Model (LLM)-powered agents demonstrate strong capabilities in autonomous task execution, tool use, and multi-step reasoning.<n> adversarial interactions can manipulate agent behavior through direct prompt injection, indirect content attacks, and multi-turn escalation strategies.<n>We propose a low-latency fraud detection layer for detecting adversarial interaction patterns in LLM-powered agents.
  arXiv  Detail & Related papers  (2026-05-01T22:35:59Z)
• From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench [11.985605415045734]
  We introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents.<n>By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing.<n>Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities.
  arXiv  Detail & Related papers  (2026-04-16T14:06:30Z)
• ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models [51.06179253173491]
  We introduce ProactiveBench, a benchmark that tests proactiveness across different tasks.<n>We evaluate 22 MLLMs on ProactiveBench, showing that (i) they generally lack proactiveness; (ii) proactiveness does not correlate with model capacity; (iii) "hinting" at proactiveness yields only marginal gains.
  arXiv  Detail & Related papers  (2026-03-19T20:58:03Z)
• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization [61.641777037967366]
  Proactive large language model (LLM) agents aim to actively plan, query, and interact over multiple turns.<n>Agentic reinforcement learning (RL) has emerged as a promising solution for training such agents in multi-turn settings.<n>We propose BAO, an agentic RL framework that combines behavior enhancement to enrich proactive reasoning and information-gathering capabilities.
  arXiv  Detail & Related papers  (2026-02-11T20:40:43Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• Dynamic benchmarking framework for LLM-based conversational data capture [0.0]
  This paper introduces a benchmarking framework to assess large language models (LLMs)<n>It integrates generative agent simulation to evaluate performance on key dimensions: information extraction, context awareness, and adaptive engagement.<n>Results show that adaptive strategies improve data extraction accuracy, especially when handling ambiguous responses.
  arXiv  Detail & Related papers  (2025-02-04T15:47:47Z)
• R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models [50.19174067263255]
  We introduce prior preference learning techniques and self-revision schedules to help the agent excel in sparse-reward, continuous action, goal-based robotic control POMDP environments. We show that our agents offer improved performance over state-of-the-art models in terms of cumulative rewards, relative stability, and success rate.
  arXiv  Detail & Related papers  (2024-09-21T18:32:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.