Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment

• URL: http://arxiv.org/abs/2603.16889v1
• Date: Sat, 28 Feb 2026 11:08:24 GMT
• Title: Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment
• Abstract summary: We introduce a rubric-guided reasoning framework that explicitly encodes multi-aspect human assessment criteria.<n>We develop an uncertainty-calibrated regression approach supported by conformal calibration for interpretable confidence intervals.<n>Our results demonstrate that rubric-guided, uncertainty-calibrated reasoning offers a principled path toward trustworthy and explainable SpeechLLM-based speech assessment.
• Score: 8.300738063140129
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliable and interpretable automated assessment of second-language (L2) speech remains a central challenge, as large speech-language models (SpeechLLMs) often struggle to align with the nuanced variability of human raters. To address this, we introduce a rubric-guided reasoning framework that explicitly encodes multi-aspect human assessment criteria: accuracy, fluency, and prosody, while calibrating model uncertainty to capture natural rating variability. We fine-tune the Qwen2-Audio-7B-Instruct model using multi-rater human judgments and develop an uncertainty-calibrated regression approach supported by conformal calibration for interpretable confidence intervals. Our Gaussian uncertainty modeling and conformal calibration approach achieves the strongest alignment with human ratings, outperforming regression and classification baselines. The model reliably assesses fluency and prosody while highlighting the inherent difficulty of assessing accuracy. Together, these results demonstrate that rubric-guided, uncertainty-calibrated reasoning offers a principled path toward trustworthy and explainable SpeechLLM-based speech assessment.

Related papers

• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• Retrieval-Augmented Linguistic Calibration [57.41519309308438]
  We model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct.<n>We present Retrieval-Augmented Linguistic truth (RALC), a lightweight post-hoc pipeline that propagates confidence signals back into natural language.
  arXiv  Detail & Related papers  (2026-05-19T04:31:38Z)
• Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis [57.78407973423517]
  We advocate uncertainty-aware subjectivity analysis, where models are expected to make predictions while expressing uncertainty that reflects human disagreement.<n>To operationalize this perspective, we propose a two-phase Disagreement Perception and Uncertainty Alignment framework.<n> Experiments on three subjectivity analysis tasks show that DPUA preserves task performance while better aligning model uncertainty with human disagreement.
  arXiv  Detail & Related papers  (2026-05-11T11:52:58Z)
• Sensitivity Uncertainty Alignment in Large Language Models [0.0]
  We propose a framework for analyzing failures of large language models under adversarial and ambiguous inputs.<n>We define a scalar score, SUA_theta(x), capturing the difference between distributional sensitivity and predictive entropy.<n>We introduce SUA-TR, a training method combining consistency regularization and entropy alignment, along with an abstention rule for safer inference.
  arXiv  Detail & Related papers  (2026-04-21T17:53:12Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness [4.129847064263056]
  We systematically evaluate the performance of Large Language Models for rubric-based short-answer grading.<n>We find that alignment is strong for binary tasks but degrades with increased rubric granularity.<n>Experiments reveal that while the model is resilient to prompt injection, it is sensitive to synonym substitutions.
  arXiv  Detail & Related papers  (2025-12-21T05:22:04Z)
• Measuring Language Model Hallucinations Through Distributional Correctness [7.106986689736826]
  A novel evaluation metric, the Distributional Correctness Score (DCS), is introduced to solve this problem.<n>DCS distinguishes between harmful overconfidence in wrong answers and uncertainty expressed through abstention, providing scores in an interpretable default range.<n>DCS is demonstrated to offer a more nuanced and aligned evaluation paradigm that incentivises models to express genuine uncertainty rather than guessing.
  arXiv  Detail & Related papers  (2025-10-05T17:50:42Z)
• Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation [20.726685669562496]
  Hallucinations are a common issue that undermine the reliability of large language models (LLMs)<n>Recent studies have identified a subset of hallucinations, known as confabulations, which arise due to predictive uncertainty of LLMs.<n>To detect confabulations, various methods for estimating predictive uncertainty in natural language generation (NLG) have been developed.
  arXiv  Detail & Related papers  (2025-10-02T17:54:09Z)
• Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models [0.0]
  We evaluate a collection of inference-time uncertainty measures to determine how closely they align with both human group-level uncertainty and traditional notions of model calibration.<n>We find that numerous measures show evidence of strong alignment to human uncertainty, even despite the lack of alignment to human answer preference.
  arXiv  Detail & Related papers  (2025-08-11T17:22:45Z)
• T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation [60.620408007636016]
  We propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores.<n>Our approach integrates Group Relative Policy Optimization into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains.
  arXiv  Detail & Related papers  (2025-05-23T13:44:59Z)
• Calibrated Large Language Models for Binary Question Answering [49.1574468325115]
  A well-calibrated model should produce probabilities that accurately reflect the likelihood of its predictions being correct. We propose a novel approach that utilizes the inductive Venn--Abers predictor (IVAP) to calibrate the probabilities associated with the output tokens corresponding to the binary labels.
  arXiv  Detail & Related papers  (2024-07-01T09:31:03Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.