ChatUMM: Robust Context Tracking for Conversational Interleaved Generation

• URL: http://arxiv.org/abs/2602.06442v1
• Date: Fri, 06 Feb 2026 07:11:50 GMT
• Title: ChatUMM: Robust Context Tracking for Conversational Interleaved Generation
• Authors: Wenxun Dai, Zhiyuan Zhao, Yule Zhong, Yiji Cheng, Jianwei Zhang, Linqing Wang, Shiyi Zhang, Yunlong Lin, Runze He, Fellix Song, Wayne Zhuang, Yong Liu, Haoji Zhang, Yansong Tang, Qinglin Lu, Chunyu Wang,
• Abstract summary: Unified multimodal models (UMMs) have achieved remarkable progress yet remain constrained by a single-turn interaction paradigm.<n>We present ChatUMM, a conversational unified model that excels at robust context tracking to sustain interleaved multimodal generation.<n>ChatUMM derives its capabilities from an interleaved multi-turn training strategy that models serialized text-image streams as a continuous conversational flow.
• Score: 44.19929499646892
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified multimodal models (UMMs) have achieved remarkable progress yet remain constrained by a single-turn interaction paradigm, effectively functioning as solvers for independent requests rather than assistants in continuous dialogue. To bridge this gap, we present ChatUMM. As a conversational unified model, it excels at robust context tracking to sustain interleaved multimodal generation. ChatUMM derives its capabilities from two key innovations: an interleaved multi-turn training strategy that models serialized text-image streams as a continuous conversational flow, and a systematic conversational data synthesis pipeline. This pipeline transforms a diverse set of standard single-turn datasets into fluid dialogues through three progressive stages: constructing basic stateful dialogues, enforcing long-range dependency resolution via ``distractor'' turns with history-dependent query rewriting, and synthesizing naturally interleaved multimodal responses. Extensive evaluations demonstrate that ChatUMM achieves state-of-the-art performance among open-source unified models on visual understanding and instruction-guided editing benchmarks, while maintaining competitive fidelity in text-to-image generation. Notably, ChatUMM exhibits superior robustness in complex multi-turn scenarios, ensuring fluid, context-aware dialogues.

Related papers

• U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation [48.6868174403074]
  We introduce U-Mind, the first unified system for high-intelligence multimodal dialogue.<n>It supports real-time generation and jointly models language, speech, motion, and video synthesis within a single interactive loop.<n>We show that U-Mind achieves state-of-the-art performance on a range of multimodal interaction tasks.
  arXiv  Detail & Related papers  (2026-02-27T07:07:02Z)
• BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration [56.98981194478512]
  We propose a unified framework that handles a broad range of subject-to-video scenarios.<n>We introduce an MLLM-DiT framework in which a pretrained multimodal large language model performs deep cross-modal reasoning to ground entities.<n>Experiments on the OpenS2V benchmark demonstrate that our method achieves superior performance across subject consistency, naturalness, and text relevance in generated videos.
  arXiv  Detail & Related papers  (2025-10-01T02:41:11Z)
• DA-Mamba: Dialogue-aware selective state-space model for multimodal engagement estimation [15.106664911098882]
  DA-Mamba is a dialogue-aware multimodal architecture that replaces attention-heavy dialogue encoders with Mamba-based selective state-space processing.<n>We show that DA-Mamba surpasses prior state-of-the-art (SOTA) methods in concordance correlation coefficient (CCC)<n>These gains enable processing much longer sequences and facilitate real-time deployment in resource-constrained, multi-party conversational settings.
  arXiv  Detail & Related papers  (2025-09-22T12:48:42Z)
• F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model [18.21432204057241]
  Fine-grained Fragment Retrieval (FFR) is a task requiring models to locate query-relevant fragments.<n>F2RVLM is a generative retrieval model trained in a two-stage paradigm.<n>It outperforms popular Vision-Language Models in both in-domain and real-domain settings.
  arXiv  Detail & Related papers  (2025-08-25T06:42:47Z)
• Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model [43.533849239738394]
  Stream- Omni is a large language-vision-speech model with efficient modality alignments.<n>It can simultaneously support interactions under various modality combinations.<n>It achieves strong performance on visual understanding, speech interaction, and vision-grounded speech interaction tasks.
  arXiv  Detail & Related papers  (2025-06-16T16:06:45Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [53.7173034249361]
  End-to-end GPT-based model OmniFlatten capable of effectively modeling complex behaviors inherent natural conversations with low latency.<n>Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full- spoken dialogue systems.
  arXiv  Detail & Related papers  (2024-10-23T11:58:58Z)
• Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking [57.92608483099916]
  Large pre-trained language models (PLMs) have shown impressive unaided performance across many NLP tasks. For more complex tasks such as dialogue state tracking (DST), designing prompts that reliably convey the desired intent is nontrivial. We introduce a saliency model to limit dialogue text length, allowing us to include more exemplars per query.
  arXiv  Detail & Related papers  (2023-02-12T15:05:10Z)
• An Exploratory Study on Long Dialogue Summarization: What Works and What's Next [33.1899354772074]
  We study long dialogue summarization by investigating three strategies to deal with the lengthy input problem and locate relevant information. Our experimental results on three long dialogue datasets (QMSum, MediaSum, SummScreen) show that the retrieve-then-summarize pipeline models yield the best performance.
  arXiv  Detail & Related papers  (2021-09-10T01:38:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.