DiffScore: Text Evaluation Beyond Autoregressive Likelihood

• URL: http://arxiv.org/abs/2605.11601v1
• Date: Tue, 12 May 2026 06:27:36 GMT
• Title: DiffScore: Text Evaluation Beyond Autoregressive Likelihood
• Abstract summary: We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models.<n>By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias.<n>Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings.
• Score: 79.18228156703695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional PMI decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings. The code is released at: https://github.com/wenlai-lavine/DiffScore.

Related papers

• XQDT: eXplainable and Quantitative Data-Text Alignment Metric with Feedback Signals [8.283940114367677]
  We present an end-to-end explainable evaluation metric that fine-tunes a language model to identify omitted, extra, incorrect, and correct data units in a data-text pair.<n>Our models outperform LLM-as-Judge methods in error prediction and achieve competitive precision, recall, and F1 scores.
  arXiv  Detail & Related papers  (2026-08-30T18:20:28Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics [25.374192139098284]
  We study prototypicality bias as a systematic failure mode in multimodal evaluation.<n>We introduce a controlled contrastive benchmark ProtoBias, spanning Animals, Objects, and Demography images.<n>Our results show that widely used metrics, including CLIPScore, PickScore, and VQA-based scores, frequently misrank these pairs.<n>We propose ProtoScore, a robust 7B- parameter metric that substantially reduces failure rates and suppresses misranking.
  arXiv  Detail & Related papers  (2026-01-08T13:49:14Z)
• Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution [27.411948623919226]
  We leverage Multi-modal Large Language Models (MLLMs) to regress accurate quality scores.<n>A key challenge is that the quality score is inherently continuous, typically modeled as a Gaussian distribution, whereas MLLMs generate discrete token outputs.<n>We propose a distribution-based approach that discretizes the score distribution into a soft label.<n>This method preserves the characteristics of the score distribution, achieving high accuracy and maintaining inter-image relationships.
  arXiv  Detail & Related papers  (2025-01-20T16:04:57Z)
• ListConRanker: A Contrastive Text Reranker with Listwise Encoding [27.017035527335402]
  We propose a novel Listwise-encoded Contrastive text reRanker (ListConRanker)<n>It can help the passage to be compared with other passages during the encoding process.<n>It achieves state-of-the-art performance on the reranking benchmark of Chinese Massive Text Embedding Benchmark.
  arXiv  Detail & Related papers  (2025-01-13T07:51:46Z)
• Generating Benchmarks for Factuality Evaluation of Language Models [61.69950787311278]
  We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation.
  arXiv  Detail & Related papers  (2023-07-13T17:14:38Z)
• Evaluating Factual Consistency of Texts with Semantic Role Labeling [3.1776833268555134]
  We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. A final factuality score is computed by an adjustable scoring mechanism. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T17:59:42Z)
• Toward Human-Like Evaluation for Natural Language Generation with Error Analysis [93.34894810865364]
  Recent studies show that considering both major errors (e.g. mistranslated tokens) and minor errors can produce high-quality human judgments. This inspires us to approach the final goal of the evaluation metrics (human-like evaluations) by automatic error analysis. We augment BARTScore by incorporating the human-like error analysis strategies, namely BARTScore++, where the final score consists of both the evaluations of major errors and minor errors.
  arXiv  Detail & Related papers  (2022-12-20T11:36:22Z)
• On the Blind Spots of Model-Based Evaluation Metrics for Text Generation [79.01422521024834]
  We explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics. We design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics.
  arXiv  Detail & Related papers  (2022-12-20T06:24:25Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application [63.10266319378212]
  We propose a method for measuring complex variables on a continuous, interval spectrum by combining supervised deep learning with the Constructing Measures approach to faceted Rasch item response theory (IRT) We demonstrate this new method on a dataset of 50,000 social media comments sourced from YouTube, Twitter, and Reddit and labeled by 11,000 U.S.-based Amazon Mechanical Turk workers.
  arXiv  Detail & Related papers  (2020-09-22T02:15:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.