Human-Anchored Factuality Evaluation with Strategic Annotation

• URL: http://arxiv.org/abs/2609.00494v1
• Date: Mon, 31 Aug 2026 23:50:10 GMT
• Title: Human-Anchored Factuality Evaluation with Strategic Annotation
• Abstract summary: LLM-based factuality judges provide scalable evaluation signals, but their metrics are often systematically biased relative to human judgments.<n>We study human-anchored factuality evaluation under limited annotation budgets, where judge predictions on the full dataset are combined with human labels on a small selectively sampled subset to obtain statistically valid estimates.<n>Our FSA-guided policy improves annotation efficiency over uniform sampling and uncertainty-driven baselines, achieving effective-sample-size gains of 40.3% on AutoFA and 27.1% on RAGTruth.
• Score: 5.771467270404003
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: LLM-based factuality judges provide scalable evaluation signals, but their metrics are often systematically biased relative to human judgments. We study human-anchored factuality evaluation under limited annotation budgets, where judge predictions on the full dataset are combined with human labels on a small selectively sampled subset to obtain statistically valid estimates. The efficiency of this approach depends critically on which examples receive human annotation: in factuality evaluation, judge-human misalignment is not driven solely by low confidence, but also by structured failure modes such as incomplete evidence, temporal mismatch, unverifiable claims, and rubric misalignment. To exploit this structure, we introduce a factuality-specific annotation policy design pipeline that uses failure-space analysis (FSA) to derive diverse predictive signals for modeling human-judge misalignment. On an internal reference-based factuality evaluation system (AutoFA) and RAGTruth, where judge-predicted estimates substantially underestimate human-annotated factual accuracy, our FSA-guided policy improves annotation efficiency over uniform sampling and uncertainty-driven baselines, achieving effective-sample-size gains of 40.3% on AutoFA and 27.1% on RAGTruth.

Related papers

• Rethinking the Evaluation and Optimization of LLM-Based Social Simulation [56.18577997572273]
  We use the subjectivity coefficient to analyze how accuracy-based evaluation and hard-label training fail as subjectivity grows.<n>We construct SUBJSIM, a benchmark of 19,300 contexts covering 193 annotators and 100 subjective questions.<n>Results on SUBJSIM demonstrate the advantages of our method.
  arXiv  Detail & Related papers  (2026-08-20T06:26:03Z)
• Position: Evaluation Scores Are Perishable Knowledge Claims [0.0]
  We argue that evaluation scores should be treated as claims with three properties: formality, scope, and validity windows.<n>We illustrate the cost of mean aggregation on the public HELM leaderboard.
  arXiv  Detail & Related papers  (2026-07-28T18:50:39Z)
• Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings [8.934085986179019]
  Shapley values are a cornerstone of explainable AI, yet their proliferation into competing formulations has created a fragmented landscape.<n>We use a unified amortized framework to isolate semantic differences between eight Shapley variants.<n>Our results reveal a fundamental misalignment: standard quantitative metrics, such as sparsity and faithfulness, are decoupled from human-perceived clarity and decision utility.
  arXiv  Detail & Related papers  (2026-04-24T15:38:44Z)
• Quantifying and Mitigating Self-Preference Bias of LLM Judges [5.017055831476582]
  Self-Preference Bias (SPB) is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation.<n>Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems.<n>We introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences.
  arXiv  Detail & Related papers  (2026-04-24T09:46:22Z)
• Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas [31.16720541398267]
  We propose a doubly-robust estimation framework designed to address evaluation sampling bias.<n>Key to our approach is the use of "persona" ratings produced by prompting an evaluator to behave as a human rater.<n>We show that our approach yields valid system quality estimates when either (i) a model trained to predict human ratings using persona ratings and source data observed under sampling bias, or (ii) a reweighting model that corrects for sampling bias is of sufficient quality.
  arXiv  Detail & Related papers  (2025-09-26T21:42:51Z)
• Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness [49.35494016290887]
  We show that equal performance across subgroups is an unreliable measure of fairness when data are representative of relevant populations but reflective of real-world disparities.<n>Our framework suggests complementing disaggregated evaluations with explicit causal assumptions and analysis to control for confounding and distribution shift.
  arXiv  Detail & Related papers  (2025-06-04T17:40:31Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation [13.325600043256552]
  We propose a semi-offline evaluation framework, where human users provide annotations of unobserved counterfactual trajectories. Our framework, combined with principled human-centered design of annotation solicitation, can enable the application of reinforcement learning in high-stakes domains.
  arXiv  Detail & Related papers  (2023-10-26T04:41:19Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation [136.16507050034755]
  Existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale. We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units. We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems.
  arXiv  Detail & Related papers  (2022-12-15T17:26:05Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.