Front-Loaded or Balanced? The Mechanism through Which Review Order Affects Overall Ratings in Premium Service Settings

• URL: http://arxiv.org/abs/2602.00008v1
• Date: Tue, 25 Nov 2025 03:12:30 GMT
• Title: Front-Loaded or Balanced? The Mechanism through Which Review Order Affects Overall Ratings in Premium Service Settings
• Authors: He Wang, Ziyu Zhou, Hanxiang Liu,
• Abstract summary: This research reveals the psychological mechanisms through which evaluation order affects consumer ratings via cognitive and affective pathways.<n>Three experiments demonstrate that in high-quality service contexts, a rating-first (vs. review-first) interface significantly elevates consumers' overall ratings.
• Score: 10.304137762077897
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In the increasingly prevalent landscape of high-quality service contexts, whether consumer evaluation interfaces adopt a rating-first or review-first sequence has become a critical factor shaping rating authenticity and feedback quality. While prior research has primarily examined review content and sentiment, systematic investigation into how evaluation order influences rating outcomes remains limited. Through exploratory analyses, we find that Letterboxd -- which employs a review-first, rating-after mechanism -- exhibits a more centralized rating distribution with fewer extreme scores, whereas Yelp -- which adopts a rating-first, review-after mechanism -- shows a pronounced bimodal distribution with more polarized ratings. Three controlled experiments further demonstrate that in high-quality service contexts, a rating-first (vs. review-first) interface significantly elevates consumers' overall ratings. Mechanism analyses indicate that cognitive effort and affective heuristics serve as dual pathways: a rating-first (vs. review-first) sequence reduces cognitive effort and heightens affective heuristics, thereby increasing rating scores. Moreover, service quality moderates this process. When service quality is low, the rating-first (vs. review-first) sequence instead leads to lower ratings. This research reveals the psychological mechanisms through which evaluation order affects consumer ratings via cognitive and affective pathways. It extends theoretical understanding of online rating formation and offers practical implications for optimizing platform interface design to enhance rating authenticity and credibility.

Related papers

• Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time [55.756345497678204]
  We introduce a new framework for evidence-based comparative study of review quality.<n>We apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL.<n>We study the relationships between measurements of review quality, and its evolution over time.
  arXiv  Detail & Related papers  (2026-01-21T16:48:29Z)
• Insights from the ICLR Peer Review and Rebuttal Process [11.96840383745149]
  We present a large-scale analysis of the ICLR 2024 and 2025 peer review processes.<n>We examine review scores, author-reviewer engagement, temporal patterns in review submissions, and co-reviewer influence effects.<n>Our findings show that initial scores and the ratings of co-reviewers are the strongest predictors of score changes during the rebuttal.
  arXiv  Detail & Related papers  (2025-11-19T14:21:52Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• SCAN: Structured Capability Assessment and Navigation for LLMs [54.54085382131134]
  textbfSCAN (Structured Capability Assessment and Navigation) is a practical framework that enables detailed characterization of Large Language Models.<n>SCAN incorporates four key components:.<n>TaxBuilder, which extracts capability-indicating tags from queries to construct a hierarchical taxonomy;.<n>RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag;.<n>A PC$2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach achieves significantly higher accuracy compared to classic LLM-as-a-Judge method
  arXiv  Detail & Related papers  (2025-05-10T16:52:40Z)
• Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards [2.8239108914343305]
  This paper argues for the need to transform the traditional one-way review system into a bi-directional feedback loop.<n>Authors evaluate review quality and reviewers earn formal accreditation, creating an accountability framework.
  arXiv  Detail & Related papers  (2025-05-08T05:51:48Z)
• Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review [4.35783648216893]
  Traditional closed peer review systems are slow, costly, non-transparent, and possibly subject to biases.<n>We propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process.
  arXiv  Detail & Related papers  (2025-01-22T17:00:27Z)
• Analytical and Empirical Study of Herding Effects in Recommendation Systems [72.6693986712978]
  We study how to manage product ratings via rating aggregation rules and shortlisted representative reviews. We show that proper recency aware rating aggregation rules can improve the speed of convergence in Amazon and TripAdvisor.
  arXiv  Detail & Related papers  (2024-08-20T14:29:23Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• On Faithfulness and Coherence of Language Explanations for Recommendation Systems [8.143715142450876]
  This work probes state-of-the-art models and their review generation component. We show that the generated explanations are brittle and need further evaluation before being taken as literal rationales for the estimated ratings.
  arXiv  Detail & Related papers  (2022-09-12T17:00:31Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• SIFN: A Sentiment-aware Interactive Fusion Network for Review-based Item Recommendation [48.1799451277808]
  We propose a Sentiment-aware Interactive Fusion Network (SIFN) for review-based item recommendation. We first encode user/item reviews via BERT and propose a light-weighted sentiment learner to extract semantic features of each review. Then, we propose a sentiment prediction task that guides the sentiment learner to extract sentiment-aware features via explicit sentiment labels.
  arXiv  Detail & Related papers  (2021-08-18T08:04:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.