Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges

• URL: http://arxiv.org/abs/2603.17172v1
• Date: Tue, 17 Mar 2026 22:08:06 GMT
• Title: Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges
• Abstract summary: Large language models (LLMs) are increasingly used as automated judges and synthetic labelers.<n>We propose a practical calibration protocol based on controlled input interventions.<n>We operationalize this with a slope-based hypothesis test over repeated trials.
• Score: 7.061518504123362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used as automated judges and synthetic labelers, especially in low-label settings. Yet these systems are stochastic and often overconfident, which makes deployment decisions difficult when external ground truth is limited. We propose a practical calibration protocol based on controlled input interventions: if noise severity increases, task performance should exhibit a statistically significant deterioration trend. We operationalize this with a slope-based hypothesis test over repeated trials, using signal-to-noise-ratio (SNR) perturbations for tabular data and lexical perturbations for text data. Across UCI tabular benchmarks and four text classification datasets, we find clear modality-dependent behavior. Our results reveal a modality gap: while text-based judges degrade predictably, the majority of tabular datasets show a lack of statistically significant performance deterioration even under significant signal-to-noise reduction. Interestingly we find that model performance is lower on datasets that are insensitive to noise interventions. We present a reproducible methodology and reporting protocol for robust LLM-judge calibration under distribution shift.

Related papers

• Robust Self-Training with Closed-loop Label Correction for Learning from Noisy Labels [29.68163073890278]
  Training deep neural networks with noisy labels remains a significant challenge.<n>We propose a self-training label correction framework using decoupled bilevel optimization.<n>Our method employs noisy posterior simulation and intermediate features to transfer ground-truth knowledge.
  arXiv  Detail & Related papers  (2026-03-14T11:10:17Z)
• NCSAM Noise-Compensated Sharpness-Aware Minimization for Noisy Label Learning [5.810900591128541]
  This paper establishes a theoretical analysis of the relationship between flatness of the loss landscape and the presence of label noise.<n>We propose Noise-Compensated Sharpness-aware Minimization (NCSAM) to leverage the perturbation of Sharpness-Aware Minimization (SAM) to remedy the damage of label noises.
  arXiv  Detail & Related papers  (2026-01-24T11:10:29Z)
• Hard vs. Noise: Resolving Hard-Noisy Sample Confusion in Recommender Systems via Large Language Models [4.7341002297388295]
  Implicit feedback, employed in training recommender systems, unavoidably confronts noise due to factors such as misclicks and position bias.<n>Previous studies have attempted to identify noisy samples through their diverged data patterns, such as higher loss values.<n>We observed that noisy samples and hard samples display similar patterns, leading to hard-noisy confusion issue.
  arXiv  Detail & Related papers  (2025-11-10T16:51:03Z)
• Modest-Align: Data-Efficient Alignment for Vision-Language Models [67.48633659305592]
  Cross-modal alignment models often suffer from overconfidence and degraded performance when operating in resource-constrained settings.<n>We propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency.<n>Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.
  arXiv  Detail & Related papers  (2025-10-24T16:11:10Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• ROPO: Robust Preference Optimization for Large Language Models [59.10763211091664]
  We propose an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods.
  arXiv  Detail & Related papers  (2024-04-05T13:58:51Z)
• Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection [82.43311784594384]
  Real-world datasets contain not only noisy labels but also class imbalance. We propose a simple yet effective method to address noisy labels in imbalanced datasets.
  arXiv  Detail & Related papers  (2024-02-17T10:34:53Z)
• Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances [55.37242480995541]
  We propose to denoise noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.
  arXiv  Detail & Related papers  (2023-10-25T17:23:37Z)
• Systematic analysis of the impact of label noise correction on ML Fairness [0.0]
  We develop an empirical methodology to evaluate the effectiveness of label noise correction techniques in ensuring the fairness of models trained on biased datasets. Our results suggest that the Hybrid Label Noise Correction method achieves the best trade-off between predictive performance and fairness.
  arXiv  Detail & Related papers  (2023-06-28T08:08:14Z)
• Open-set Label Noise Can Improve Robustness Against Inherent Label Noise [27.885927200376386]
  We show that open-set noisy labels can be non-toxic and even benefit the robustness against inherent noisy labels. We propose a simple yet effective regularization by introducing Open-set samples with Dynamic Noisy Labels (ODNL) into training.
  arXiv  Detail & Related papers  (2021-06-21T07:15:50Z)
• Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model [80.91927573604438]
  This paper proposes a simple yet universal probabilistic model, which explicitly relates noisy labels to their instances. Experiments on datasets with both synthetic and real-world label noise verify that the proposed method yields significant improvements on robustness.
  arXiv  Detail & Related papers  (2021-01-14T05:43:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.