TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

• URL: http://arxiv.org/abs/2603.16759v1
• Date: Tue, 17 Mar 2026 16:40:35 GMT
• Title: TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities
• Abstract summary: We introduce a new benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation.<n>We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data.<n>Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance.
• Score: 57.865325277036455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-turn conversations are a common and critical mode of language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval.

Related papers

• TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue [79.83228182492354]
  Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor.<n>We present TurnBench, a benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection.<n>We find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles.
  arXiv  Detail & Related papers  (2026-08-25T23:14:36Z)
• Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap [51.23637310487053]
  We introduce Found in Conversation, a training framework where a model teaches itself to find and recover its single-turn competence.<n>View-Asymmetric Self-Distillation distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior.<n>Across model families (Llama, Qwen, Phi, and OLMo), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones.
  arXiv  Detail & Related papers  (2026-05-23T06:58:52Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues [59.12542274007847]
  LoopServe is an adaptive dual-phase inference framework for large language models in multi-turn dialogues.<n>LoopServe consistently achieves superior effectiveness compared to existing baselines.
  arXiv  Detail & Related papers  (2025-07-18T06:12:08Z)
• Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling [13.628984890958314]
  We introduce a data-centric customization approach for efficiently enhancing multimodal understanding in conversational speech modeling.<n>Our approach achieves state-of-the-art performance on the Spoken-SQuAD benchmark, using only 10% of the training data with open-weight models.<n>We also introduce ASK-QA, the first dataset for multi-turn spoken dialogue with ambiguous user requests and dynamic evaluation inputs.
  arXiv  Detail & Related papers  (2024-12-20T15:43:09Z)
• MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models [70.92847554971065]
  We introduce MT-Eval, a comprehensive benchmark designed to evaluate multi-turn conversational abilities. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. Our evaluation of 11 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks.
  arXiv  Detail & Related papers  (2024-01-30T04:50:28Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• MultiTurnCleanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup [13.4369374707379]
  This study proposes an innovative Multi-Turn Cleanup task for spoken conversational transcripts. We design a data labeling schema to collect the high-quality dataset and provide extensive data analysis.
  arXiv  Detail & Related papers  (2023-05-19T22:50:02Z)
• One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech [3.42658286826597]
  We introduce an approach to multilingual speech synthesis which uses the meta-learning concept of contextual parameter generation. Our model is shown to effectively share information across languages and according to a subjective evaluation test, it produces more natural and accurate code-switching speech than the baselines.
  arXiv  Detail & Related papers  (2020-08-03T10:43:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.