Interpreting Language Reward Models via Contrastive Explanations

• URL: http://arxiv.org/abs/2411.16502v1
• Date: Mon, 25 Nov 2024 15:37:27 GMT
• Title: Interpreting Language Reward Models via Contrastive Explanations
• Abstract summary: Reward models (RMs) are a crucial component in the alignment of large language models' (LLMs) outputs with human values. We propose to use contrastive explanations to explain any binary response comparison made by an RM.
• Score: 14.578645682339983
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward models (RMs) are a crucial component in the alignment of large language models' (LLMs) outputs with human values. RMs approximate human preferences over possible LLM responses to the same prompt by predicting and comparing reward scores. However, as they are typically modified versions of LLMs with scalar output heads, RMs are large black boxes whose predictions are not explainable. More transparent RMs would enable improved trust in the alignment of LLMs. In this work, we propose to use contrastive explanations to explain any binary response comparison made by an RM. Specifically, we generate a diverse set of new comparisons similar to the original one to characterise the RM's local behaviour. The perturbed responses forming the new comparisons are generated to explicitly modify manually specified high-level evaluation attributes, on which analyses of RM behaviour are grounded. In quantitative experiments, we validate the effectiveness of our method for finding high-quality contrastive explanations. We then showcase the qualitative usefulness of our method for investigating global sensitivity of RMs to each evaluation attribute, and demonstrate how representative examples can be automatically extracted to explain and compare behaviours of different RMs. We see our method as a flexible framework for RM explanation, providing a basis for more interpretable and trustworthy LLM alignment.

Related papers

• Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study [13.666045109234107]
  We analyze BERTScore and SemDist across different Large Language Models (LLMs) for Automatic Speech Recognition (ASR) evaluation.<n>Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.
  arXiv  Detail & Related papers  (2026-08-26T09:36:48Z)
• Deliberate Evolution: Agentic Reasoning for Sample-Efficient Symbolic Regression with LLMs [52.89778838903305]
  Deliberate Evolution (DE) is an agentic framework that decouples symbolic generation from search control.<n>Experiments on LLM-SRBench show that DE consistently outperforms representative LLM-based SR baselines across diverse scientific domains.
  arXiv  Detail & Related papers  (2026-06-03T02:22:16Z)
• Understanding Generalization in Role-Playing Models via Information Theory [78.78610491091096]
  We introduce an information-theoretic metric, named reasoning-based effective mutual information difference (R-EMID), to measure RPM performance degradation.<n>We also propose a co-evolving reinforcement learning framework to adaptively model the connection among user, character, and dialogue context.
  arXiv  Detail & Related papers  (2025-12-19T06:37:44Z)
• SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder [54.31950189922548]
  Reward models (RMs) are proxies for human preference evaluation and guiding model alignment.<n>We propose SparseRM, which leverages Sparse Autoencoder (SAE) to extract preference-relevant information encoded in model representations.<n>SparseRM achieves superior performance over most mainstream RMs while using less than 1% of trainable parameters.
  arXiv  Detail & Related papers  (2025-11-11T06:51:56Z)
• MADREC: A Multi-Aspect Driven LLM Agent for Explainable and Adaptive Recommendation [11.430206422495829]
  Multi-Aspect Driven LLM Agent MADRec is an autonomous recommender that constructs user and item profiles by unsupervised extraction of multi-aspect information from reviews.<n>MADRec generates structured profiles via aspect-category-based summarization and applies Re-Ranking to construct high-density inputs.<n>Experiments across multiple domains show that MADRec outperforms traditional and LLM-based baselines in both precision and explainability.
  arXiv  Detail & Related papers  (2025-10-15T10:03:29Z)
• From latent factors to language: a user study on LLM-generated explanations for an inherently interpretable matrix-based recommender system [8.280161440212504]
  We investigate whether large language models (LLMs) can generate effective, user-facing explanations from a mathematically interpretable recommendation model.<n>We conduct a study with 326 participants who assessed the quality of the explanations across five key dimensions.<n>Our analysis reveals that all explanation types are generally well received, with moderate statistical differences between strategies.
  arXiv  Detail & Related papers  (2025-09-23T13:30:03Z)
• RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns [50.401907401444404]
  Large language models (LLMs) are crucial for preventing misuse and building trustworthy AI systems.<n>We propose RepreGuard, an efficient statistics-based detection method.<n> Experimental results show that RepreGuard outperforms all baselines with average 94.92% AUROC on both in-distribution (ID) and OOD scenarios.
  arXiv  Detail & Related papers  (2025-08-18T17:59:15Z)
• Interpretable Reward Model via Sparse Autoencoder [16.903840987027912]
  We introduce the Sparse Autoencoder-enhanced Reward Model (SARM), a novel architecture that integrates a pretrained Sparse Autoencoder into a reward model.<n>SARM maps the hidden activations of LLM-based RM into an interpretable, sparse, and monosemantic feature space.<n> Empirical evaluations demonstrate that SARM facilitates direct feature-level attribution of reward assignments, allows dynamic adjustment to preference shifts, and achieves superior alignment performance compared to conventional reward models.
  arXiv  Detail & Related papers  (2025-08-12T08:41:00Z)
• Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback [52.1410307583181]
  We useReinforcement Learning from Human Feedback to train language models (LMs) to follow complex human preferences.<n>As training progresses, the responses generated by the LM no longer resemble the responses seen by the reward model (RM)<n>We propose Off-Policy Corrected Reward Modeling to correct the RM using importance weighting, without requiring new labels or samples.
  arXiv  Detail & Related papers  (2025-07-21T11:19:04Z)
• Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations [30.68740512996253]
  Chain-of-thought explanations are widely used to inspect the decision process of large language models. We show that preference optimization can inadvertently reduce the faithfulness of these explanations.
  arXiv  Detail & Related papers  (2025-04-07T17:49:23Z)
• Like Father, Like Son: Kinship-Aware Preference Mapping (KARMA) for Automatic Alignment in Large Language Models [2.970904425631548]
  Kinship-Aware pReference MApping (KARMA) is a novel framework that pairs responses from models with comparable competencies. By constraining preference comparisons to outputs of similar complexity and quality, KARMA enhances the informativeness of preference data. Empirical evaluations demonstrate that our kinship-aware approach leads to more consistent and interpretable alignment outcomes.
  arXiv  Detail & Related papers  (2025-02-26T01:36:40Z)
• Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis [10.133537818749291]
  Large language models (LLMs) have demonstrated significant utilities in real-world applications. Benchmark evaluations are crucial for assessing the capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-02-13T03:43:33Z)
• Guide-to-Explain for Controllable Summarization [11.904090197598505]
  controllable summarization with large language models (LLMs) remains underexplored. We propose a guide-to-explain framework (GTE) for controllable summarization. Our framework enables the model to identify misaligned attributes in the initial draft and guides it in explaining errors in the previous output.
  arXiv  Detail & Related papers  (2024-11-19T12:36:02Z)
• Rationale Behind Essay Scores: Enhancing S-LLM's Multi-Trait Essay Scoring with Rationale Generated by LLMs [2.324913904215885]
  This paper introduces Rationale-based Multiple Trait Scoring (RMTS), a novel approach for multi-trait essay scoring. RMTS integrates prompt-engineering-based large language models (LLMs) with a fine-tuning-based essay scoring model using a smaller large language model (S-LLM) Experiments on benchmark datasets, including ASAP, ASAP++, and Feedback Prize, show that RMTS significantly outperforms state-of-the-art models and vanilla S-LLMs in trait-specific scoring.
  arXiv  Detail & Related papers  (2024-10-18T06:35:17Z)
• RMB: Comprehensively Benchmarking Reward Models in LLM Alignment [44.84304822376291]
  Reward models (RMs) guide the alignment of large language models (LLMs) We propose RMB, a comprehensive RM benchmark that covers over 49 real-world scenarios. Based on our benchmark, we conduct extensive analysis on the state-of-the-art RMs.
  arXiv  Detail & Related papers  (2024-10-13T16:06:54Z)
• Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment [53.17596274334017]
  We evaluate the correctness of the detailed inference patterns of an LLM behind its seemingly correct outputs.<n>Experiments show that even when the language generation results appear correct, a significant portion of the inference patterns used by the LLM for the legal judgment may represent misleading or irrelevant logic.
  arXiv  Detail & Related papers  (2024-10-06T08:33:39Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts [23.27203570485055]
  Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models with human preferences. We propose a two-stage approach to train a reward model (RM) with multi-dimensional absolute-rating data. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow on top of the ArmoRM.
  arXiv  Detail & Related papers  (2024-06-18T17:58:28Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves [57.974103113675795]
  We present a method named Rephrase and Respond' (RaR) which allows Large Language Models to rephrase and expand questions posed by humans. RaR serves as a simple yet effective prompting method for improving performance. We show that RaR is complementary to the popular Chain-of-Thought (CoT) methods, both theoretically and empirically.
  arXiv  Detail & Related papers  (2023-11-07T18:43:34Z)
• Confronting Reward Model Overoptimization with Constrained RLHF [114.71591361764547]
  We show that correlation between component RMs has a significant effect on the locations of these points. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers.
  arXiv  Detail & Related papers  (2023-10-06T16:59:17Z)
• Manifold Characteristics That Predict Downstream Task Performance [2.642698101441705]
  We show that differences between methods can be understood more clearly by investigating the representation manifold directly. We propose a framework and new metric to measure and compare different RMs. We show that self-supervised methods learn an RM where alterations lead to large but constant size changes, indicating a smoother RM than fully supervised methods.
  arXiv  Detail & Related papers  (2022-05-16T06:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.