Decoupling Scores and Text: The Politeness Principle in Peer Review

• URL: http://arxiv.org/abs/2604.14162v1
• Date: Mon, 23 Mar 2026 11:58:48 GMT
• Title: Decoupling Scores and Text: The Politeness Principle in Peer Review
• Abstract summary: Authors often struggle to interpret peer review feedback, deriving false hope from polite comments or feeling confused by specific low scores.<n>We construct a dataset of over 30,000 ICLR 2021-2025 submissions and compare acceptance prediction performance using numerical scores versus text reviews.<n>Our experiments reveal a significant performance gap: score-based models achieve 91% accuracy, while text-based models reach only 81% even with large language models.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Authors often struggle to interpret peer review feedback, deriving false hope from polite comments or feeling confused by specific low scores. To investigate this, we construct a dataset of over 30,000 ICLR 2021-2025 submissions and compare acceptance prediction performance using numerical scores versus text reviews. Our experiments reveal a significant performance gap: score-based models achieve 91% accuracy, while text-based models reach only 81% even with large language models, indicating that textual information is considerably less reliable. To explain this phenomenon, we first analyze the 9% of samples that score-based models fail to predict, finding their score distributions exhibit high kurtosis and negative skewness, which suggests that individual low scores play a decisive role in rejection even when the average score falls near the borderline. We then examine why text-based accuracy significantly lags behind scores from a review sentiment perspective, revealing the prevalence of the Politeness Principle: reviews of rejected papers still contain more positive than negative sentiment words, masking the true rejection signal and making it difficult for authors to judge outcomes from text alone.

Related papers

• Beyond the Name: Demographic Leakage in De-Identified Résumés and Evaluation Artifacts in LLM Bias Audits [4.736836688901646]
  We investigate whether eliminating language fields resolves leakage across nine open-weight models and 620 résumés.<n>We isolate unstructured prose across five ethnocultural conditions and three cue-salience tiers.<n>Target-group recovery averages 0.757 overall and saturates at 1.000 under high salience.
  arXiv  Detail & Related papers  (2026-09-15T01:48:44Z)
• Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes [0.0]
  Large language model (LLM) systems are increasingly proposed to assist peer review.<n>Most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.<n>We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score.
  arXiv  Detail & Related papers  (2026-06-14T16:13:15Z)
• SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones? [51.154921661608675]
  We introduce SoundnessBench, a curated benchmark of 1,099 machine-learning research proposals reconstructed from ICLR submissions.<n>SoundnessBench should be interpreted as a benchmark for recoverable proposal-stage soundness rather than exact prediction of full-paper review outcomes.
  arXiv  Detail & Related papers  (2026-05-28T17:57:37Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias [3.9562034587217187]
  We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates.<n>We observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions.<n>This analysis shows that LOC bias is often detectable with very small validation sets.
  arXiv  Detail & Related papers  (2026-03-31T21:40:36Z)
• Aligned Textual Scoring Rules [14.705645899416117]
  A scoring rule is proper if, from the agent's perspective, reporting the true belief maximizes the expected score.<n>Our paper designs the Aligned Scoring rule (ASR) for text by optimizing and minimizing the mean squared error between a proper scoring rule and a reference score.
  arXiv  Detail & Related papers  (2025-07-08T17:53:22Z)
• Structured Reasoning for Fairness: A Multi-Agent Approach to Bias Detection in Textual Data [0.0]
  We propose a multi-agent framework that identifies by disentangling each statement as fact or opinion.<n>By combining enhanced detection accuracy with interpretable explanations, this approach promotes accountability in modern language models.
  arXiv  Detail & Related papers  (2025-03-01T05:27:54Z)
• Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation [49.3814117521631]
  Standard benchmarks of bias and fairness in large language models (LLMs) measure the association between the user attributes stated or implied by a prompt.<n>We develop analogous RUTEd evaluations from three contexts of real-world use: children's bedtime stories, user personas, and English language learning exercises.<n>We find that standard bias metrics have no significant correlation with the more realistic bias metrics.
  arXiv  Detail & Related papers  (2024-02-20T01:49:15Z)
• Semantic Properties of cosine based bias scores for word embeddings [48.0753688775574]
  We propose requirements for bias scores to be considered meaningful for quantifying biases. We analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.
  arXiv  Detail & Related papers  (2024-01-27T20:31:10Z)
• Human Feedback is not Gold Standard [28.63384327791185]
  We critically analyse the use of human feedback for both training and evaluation. We find that while preference scores have fairly good coverage, they under-represent important aspects like factuality.
  arXiv  Detail & Related papers  (2023-09-28T11:18:20Z)
• Measuring the Effect of Influential Messages on Varying Personas [67.1149173905004]
  We present a new task, Response Forecasting on Personas for News Media, to estimate the response a persona might have upon seeing a news message. The proposed task not only introduces personalization in the modeling but also predicts the sentiment polarity and intensity of each response. This enables more accurate and comprehensive inference on the mental state of the persona.
  arXiv  Detail & Related papers  (2023-05-25T21:01:00Z)
• The SAME score: Improved cosine based bias score for word embeddings [49.75878234192369]
  We introduce SAME, a novel bias score for semantic bias in embeddings. We show that SAME is capable of measuring semantic bias and identify potential causes for social bias in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-28T09:28:13Z)
• Balancing out Bias: Achieving Fairness Through Training Reweighting [58.201275105195485]
  Bias in natural language processing arises from models learning characteristics of the author such as gender and race. Existing methods for mitigating and measuring bias do not directly account for correlations between author demographics and linguistic variables. This paper introduces a very simple but highly effective method for countering bias using instance reweighting.
  arXiv  Detail & Related papers  (2021-09-16T23:40:28Z)
• Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation [109.06060143938052]
  We propose a "double perturbation" framework to uncover model weaknesses beyond the test dataset. We apply this framework to study two perturbation-based approaches that are used to analyze models' robustness and counterfactual bias in English.
  arXiv  Detail & Related papers  (2021-04-12T06:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.