Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment

• URL: http://arxiv.org/abs/2606.01456v1
• Date: Sun, 31 May 2026 21:30:43 GMT
• Title: Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment
• Abstract summary: Large language models are increasingly deployed as advisors whose objective is not aligned with the user's.<n>We turn the canonical Crawford-Sobel cheap-talk model into a benchmark for honesty under preference misalignment.<n>We find all four over-reveal relative to the most-informative equilibrium by 1.8 to 4.2x.
• Score: 0.8699280339422538
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly deployed as advisors whose objective is not aligned with the user's: recommenders optimize for engagement, sales assistants for purchases, negotiation agents for concessions. Whether such advisors stay truthful when honesty conflicts with their own payoff is a core alignment-evaluation question. We turn the canonical Crawford-Sobel cheap-talk model into a pre-specified benchmark for LLM honesty under preference misalignment. Cheap-talk theory predicts neither full revelation nor silence but coarse monotone partitions, with fewer informative intervals as preference conflict grows. A sender observes a state omega in [0,1], wants the receiver's action near omega+b, and sends one costless message to a receiver whose ideal action is omega. The design uses 5 bias levels, 3 prompt frames, a fixed low-temperature setting, and 200 states per cell: 12,000 sender calls. For the positive-bias grid b in {0.01,0.04,0.08,0.12} the exact most-informative partition sizes are 7,4,3,2, with oracle normalized mutual information 0.5294, 0.3268, 0.2205, 0.1829. Running the full design on four instruction-tuned models (GPT-4o, Claude Sonnet 4.5, Gemini 2.5 Flash-Lite, Llama-3.3-70B), we find all four over-reveal relative to the most-informative equilibrium by 1.8 to 4.2x: normalized mutual information stays at 0.78-0.94 where the oracle prescribes 0.18-0.53. Informativeness declines with bias as predicted but never approaches the strategic optimum; rather than coarse partitions, models show near-full revelation with a constant upward offset tracking their bias (linear exaggeration). Payoff-maximizing versus honesty framing has negligible effect. A decoder ablation shows the finding is recoverable only when the receiver reads the sender's stated number: an embedding-only decoder mis-reads the same data as near-babbling.

Related papers

• SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement [2.8129261352670003]
  We present sys, a statistics-first gating architecture that treats procurement as a cost-aware routing problem.<n>Gate accepts a decision only when intervals are tight, sample sizes are adequate, and the axes agree.<n>We report honest negative diagnostics of the debate path: a con-side win rate of 0.80 and a 52% position-flip rate under advocate swapping.
  arXiv  Detail & Related papers  (2026-07-21T10:52:04Z)
• Unbiased Alignment for Large Language Models with Noisy Preferences [74.69408994904965]
  We present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss.<n>By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets.
  arXiv  Detail & Related papers  (2026-07-03T12:04:43Z)
• Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit [5.437038443632133]
  We test an implicit assumption that scale and general capability are sufficient for nuanced classification of misinformation discourse.<n>We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs, and fine-tuned DistilBERT and RoBERTa.<n>In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice.
  arXiv  Detail & Related papers  (2026-06-02T22:58:59Z)
• Persona Conditioning of Brand Recommendations in Retrieval-Augmented Commercial Chat: A Prominence-Stratified Cross-Provider Audit [0.0]
  We audit how strongly that contextual variation reshapes which brands the model recommends.<n>The audit samples 2,000 runs over a design space of 10 personas x 8 prompts x 3 model configurations x N=10 reps.<n>Any measurement of AI brand perception must condition on the buyer persona supplying the query.
  arXiv  Detail & Related papers  (2026-05-28T16:43:38Z)
• Correcting Selection Bias in Sparse User Feedback for Large Language Model Quality Estimation: A Multi-Agent Hierarchical Bayesian Approach [0.9558392439655014]
  We propose a three-agent hierarchical Bayesian pipeline that does not require ground-truth labels on individual interactions.<n>A mild prior on the feedback channel (typical positive-feedback rate and negative-to-positive ratio) keeps Hierarchical-Informed within 4-13 pp of $Qstar$ as the bias ratio sweeps.<n>Without channel-side priors, every weak-prior variant misses $Qstar$ by 22-33 pp.
  arXiv  Detail & Related papers  (2026-05-12T14:22:06Z)
• Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science [0.11280931253550518]
  We audit three open-source 7B instruction-tuned models (Zephyr, Mistral-Instruct, Qwen2.5-Instruct) across six TweetEval tasks.<n>All three models exhibit neutrality bias on abortion stance, underestimating opposition prevalence by 24 to 40 percentage points and inflating the neutral label.<n>Strikingly, Zephyr's hate-speech prevalence estimate matches the gold rate exactly while its class-conditional errors are large in both directions.
  arXiv  Detail & Related papers  (2026-05-12T08:14:10Z)
• Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines [0.0]
  We compare nine debiasing strategies across five judge models from four provider families.<n>Style bias is the dominant bias (0.76-0.92 across all models), far exceeding position bias (= 0.04)<n>The combined budget strategy significantly improves Claude Sonnet 4 by +11.2 pp (p 0.0001), with directionally positive trends for other models.
  arXiv  Detail & Related papers  (2026-04-25T07:18:30Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks [12.396822247035578]
  We present exMT, a benchmark for objective extraction and metacognition.<n>Given a multi-turn transcript, a model must output a one-sentence base objective and a self-reported confidence.<n> Accuracy is scored by similarity to gold objectives, then thresholded once on 300 calibration items.
  arXiv  Detail & Related papers  (2025-08-23T03:32:04Z)
• What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding [84.42056293290015]
  We analyze the token-level misalignment between reasoning and non-reasoning models.<n>Motivated by the Local Misalignment Diminish, we propose FoReaL-Decoding.<n>On four popular math-reasoning benchmarks, FoReaL-Decoding reduces theoretical FLOPs by 30 to 50% and trims CoT length by up to 40%.
  arXiv  Detail & Related papers  (2025-06-08T05:08:32Z)
• Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models [15.53216696218776]
  We explore the internal mechanisms of how bias emerges in large language models (LLMs) when provided with ambiguous comparative prompts. We propose $textttATLAS$, a technique to localize bias to specific layers of the LLM by analyzing attention scores and then reduce bias by scaling attention in these biased layers.
  arXiv  Detail & Related papers  (2024-10-29T20:15:56Z)
• LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models [69.68379406317682]
  We introduce a listener-aware finetuning method (LACIE) to calibrate implicit and explicit confidence markers. We show that LACIE models the listener, considering not only whether an answer is right, but whether it will be accepted by a listener. We find that training with LACIE results in 47% fewer incorrect answers being accepted while maintaining the same level of acceptance for correct answers.
  arXiv  Detail & Related papers  (2024-05-31T17:16:38Z)
• Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
  Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences. We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
  arXiv  Detail & Related papers  (2024-05-01T17:59:20Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.