Measuring Semantic Progress in Multi-turn Dialogue via Information Gain

• URL: http://arxiv.org/abs/2606.12332v1
• Date: Wed, 10 Jun 2026 17:04:59 GMT
• Title: Measuring Semantic Progress in Multi-turn Dialogue via Information Gain
• Abstract summary: We focus on a key dimension of information-seeking dialogue: semantic progress.<n>We introduce an information-theoretic metric that approximates it in embedding space.<n> Experiments show that the proposed metric achieves competitive agreement with human judgments.
• Score: 7.146946575593476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating multi-turn dialogue is challenging because quality emerges across turns rather than within individual responses. We focus on a key dimension of information-seeking dialogue: semantic progress, defined as the accumulation of new, question-relevant, and non-redundant information over the course of a conversation. We formalize semantic progress as question-conditioned uncertainty reduction and introduce an information-theoretic metric that approximates it in embedding space. Our main estimator uses a tractable Gaussian formulation with closed-form updates, while a complementary maximum-entropy argument shows why log-determinant structure arises more broadly when only second-order embedding information is retained. This formulation yields desirable theoretical properties, including monotonicity, additive decomposition of total information gain across turns, and diminishing returns for redundant evidence. Unlike LLM-as-a-judge approaches, our metric requires no autoregressive inference at evaluation time and is fully reproducible for a fixed embedding model. Experiments on MT-Bench, Chatbot Arena, and UltraFeedback show that the proposed metric achieves competitive agreement with human judgments despite targeting only semantic progress, with improved alignment on MT-Bench and UltraFeedback compared to several LLM-based judges. Notably, the method remains effective with lightweight embedding models under CPU-only execution, indicating that semantic progress can be captured without reliance on large model capacity.

Related papers

• Learning Sample-wise Rank-aware Interpolation Weights for Composed Visual Data Retrieval [39.97218342175001]
  We introduce SRAIN, the first framework that dynamically predicts query-specific weights.<n>SRAIN achieves the best in composed video retrieval and matches the current state of the art in composed image retrieval.
  arXiv  Detail & Related papers  (2026-08-23T16:38:00Z)
• CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement [50.91045324738942]
  CLAIM is an uncertainty-driven framework for active clarification learning in open-domain settings.<n>We propose an entropy-driven synthetic data generation pipeline that integrates uncertainty estimation with semantic clustering and reasoning-based judgments.<n> Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data.
  arXiv  Detail & Related papers  (2026-08-12T04:27:45Z)
• MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue [98.58433916474472]
  We introduce MM-Snowball, the first benchmark for diagnosis of hallucination snowballing within dialogues.<n>We propose Conflict-Aware Visual Rectification (CAVR) to counteract this degradation.
  arXiv  Detail & Related papers  (2026-05-30T08:53:34Z)
• Max-pooling Network Revisited: Analyzing the Role of Semantic Probability in Multiple Instance Learning for Hallucination Detection [31.935824861650005]
  Hallucination detection has become increasingly important for improving the reliability of large language models.<n>Recently, hybrid approaches such as HaMI, which combine semantic consistency with internal model states via Multiple Instance Learning (MIL), have achieved state-of-the-art performance.<n>In this work, we first provide a theoretical analysis of HaMI in terms of decision margins, revealing that scaling internal states with semantic consistency leads to an enlarged decision margin.
  arXiv  Detail & Related papers  (2026-05-09T10:30:30Z)
• LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations [53.20772659095155]
  This paper tackles the more challenging setting of IML under training-time incomplete observations.<n>We propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), a framework that reformulates this challenge as a conditional sequence reasoning task.
  arXiv  Detail & Related papers  (2026-05-01T06:11:42Z)
• Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition [36.36218470387896]
  We recast this issue as a causal inference problem and ask: Would the prediction remain if the object appeared in a different environment?<n>To answer this at inference time, we estimate object and background expectations within CLIP's representation space, and synthesize counterfactual embeddings.<n>Without retraining or prompt design, our method substantially improves both worst-group and average accuracy on context-sensitive benchmarks.
  arXiv  Detail & Related papers  (2025-10-30T13:11:23Z)
• DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models [55.30555646945055]
  Text-to-Image (T2I) models are vulnerable to semantic leakage.<n>We introduce DeLeaker, a lightweight approach that mitigates leakage by directly intervening on the model's attention maps.<n>SLIM is the first dataset dedicated to semantic leakage.
  arXiv  Detail & Related papers  (2025-10-16T17:39:21Z)
• KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues [58.305425399644086]
  Multi-Turn Long-Form Question Answering (MT-LFQA) is a key application paradigm of Large Language Models (LLMs) in knowledge-intensive domains.<n>We introduce textbfKnowMT-Bench, the textitfirst-ever benchmark designed to systematically evaluate MT-LFQA for LLMs across knowledge-intensive fields.
  arXiv  Detail & Related papers  (2025-09-26T04:32:29Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Invoke Interfaces Only When Needed: Adaptive Invocation for Large Language Models in Question Answering [8.926050258264844]
  We propose a practical invocation evaluation metric called AttenHScore.<n>It calculates the accumulation and propagation of hallucinations during the generation process of small LMs.<n>By dynamically adjusting the detection threshold, we achieve more accurate real-time invocation of large LMs.
  arXiv  Detail & Related papers  (2025-05-05T01:45:56Z)
• QR-VC: Leveraging Quantization Residuals for Linear Disentanglement in Zero-Shot Voice Conversion [10.376226860586351]
  Zero-shot voice conversion is a technique that alters the speaker identity of an input speech to match a target speaker using only a single reference utterance.<n>Recent approaches extensively utilize self-supervised learning features with K-means quantization to extract high-quality content representations.<n>We introduce a novel approach that fully utilizes quantization residuals by leveraging temporal properties of speech components.
  arXiv  Detail & Related papers  (2024-11-25T07:14:26Z)
• Elastic Weight Removal for Faithful and Abstractive Dialogue Generation [61.40951756070646]
  A dialogue system should generate responses that are faithful to the knowledge contained in relevant documents. Many models generate hallucinated responses instead that contradict it or contain unverifiable information. We show that our method can be extended to simultaneously discourage hallucinations and extractive responses.
  arXiv  Detail & Related papers  (2023-03-30T17:40:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.