Learning When to Trust in Contextual Bandits

• URL: http://arxiv.org/abs/2603.13356v1
• Date: Mon, 09 Mar 2026 01:35:37 GMT
• Title: Learning When to Trust in Contextual Bandits
• Abstract summary: We term this mode as Contextual Sycophancy, where evaluators are truthful in benign contexts but strategically biased in critical ones.<n>We propose CESA-LinUCB, which learns a high-dimensional Trust Boundary for each evaluator.<n>We prove that CESA-LinUCB achieves sublinear regret $tildeO(sqrtT)$ against contextual adversaries, recovering the ground truth even when no evaluator is globally reliable.
• Score: 1.3428344011390776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard approaches to Robust Reinforcement Learning assume that feedback sources are either globally trustworthy or globally adversarial. In this paper, we challenge this assumption and we identify a more subtle failure mode. We term this mode as Contextual Sycophancy, where evaluators are truthful in benign contexts but strategically biased in critical ones. We prove that standard robust methods fail in this setting, suffering from Contextual Objective Decoupling. To address this, we propose CESA-LinUCB, which learns a high-dimensional Trust Boundary for each evaluator. We prove that CESA-LinUCB achieves sublinear regret $\tilde{O}(\sqrt{T})$ against contextual adversaries, recovering the ground truth even when no evaluator is globally reliable.

Related papers

• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation [67.26315138466312]
  We propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts.<n>Our findings highlight that capability distillation does not imply confidence, and that confidence should be treated as an essential objective in post-training.
  arXiv  Detail & Related papers  (2026-04-18T04:43:40Z)
• FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning [49.65751420291115]
  Chain-of-Thought (CoT) prompting has improved LLM reasoning, but models often generate explanations that appear coherent while containing unfaithful intermediate steps.<n>We propose FACT-E, a causality-inspired framework for evaluating CoT quality.<n>We show that FACT-E improves reasoning-trajectory selection and yields stronger in-context learnings.
  arXiv  Detail & Related papers  (2026-04-12T15:35:08Z)
• Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majorities [1.3428344011390776]
  We show that Dogma 4 of Reinforcement Learning fails in social settings where evaluators may be sycophantic, lazy, or adversarial.<n>We propose Epistemic Source Alignment (ESA) to resolve this problem.<n>ESA utilizes sparse safety axioms to judge the source of the feedback rather than the signal itself.
  arXiv  Detail & Related papers  (2026-02-08T19:23:02Z)
• Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency [78.91846841708586]
  We show that even facts answered with perfect self-consistency can rapidly collapse under mild contextual interference.<n>We propose Neighbor-Consistency Belief (NCB), a structural measure of belief that evaluates response coherence across a conceptual neighborhood.<n>We also present Structure-Aware Training (SAT), which optimize context-invariant belief structure and reduces long-tail knowledge brittleness by approximately 30%.
  arXiv  Detail & Related papers  (2026-01-09T16:23:21Z)
• From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse [19.382948880868522]
  This study constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth.<n>We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy.<n>This suggests a feasible approach applicable in real-world settings with inherent constraints.
  arXiv  Detail & Related papers  (2025-09-18T07:57:18Z)
• Context Engineering for Trustworthiness: Rescorla Wagner Steering Under Mixed and Inappropriate Contexts [55.70338710797578]
  We introduce the Poisoned Context Testbed, pairing queries with real-world contexts containing relevant and inappropriate content.<n>Inspired by associative learning in animals, we adapt the Rescorla-Wagner (RW) model from neuroscience to quantify how competing contextual signals influence LLM outputs.<n>We introduce RW-Steering, a two-stage finetuning-based approach that enables the model to internally identify and ignore inappropriate signals.
  arXiv  Detail & Related papers  (2025-09-02T00:40:34Z)
• When to Trust Context: Self-Reflective Debates for Context Reliability [32.806602222335485]
  Self-Reflective Debate for Contextual Reliability (SR-DCR) is a lightweight framework that integrates token-level self-confidence with an asymmetric multi-agent debate to adjudicate such conflicts.<n> Experiments on the ClashEval benchmark demonstrate that SR-DCR consistently enhances robustness while maintaining accuracy on trustworthy inputs.
  arXiv  Detail & Related papers  (2025-06-06T12:09:34Z)
• Aligning Large Language Models for Faithful Integrity Against Opposing Argument [71.33552795870544]
  Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks.<n>They can be easily misled by unfaithful arguments during conversations, even when their original statements are correct.<n>We propose a novel framework, named Alignment for Faithful Integrity with Confidence Estimation.
  arXiv  Detail & Related papers  (2025-01-02T16:38:21Z)
• FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows" [74.7488607599921]
  FaithEval is a benchmark to evaluate the faithfulness of large language models (LLMs) in contextual scenarios.<n>FaithEval comprises 4.9K high-quality problems in total, validated through a rigorous four-stage context construction and validation framework.<n>Our study reveals that even state-of-the-art models often struggle to remain faithful to the given context, and that larger models do not necessarily exhibit improved faithfulness.
  arXiv  Detail & Related papers  (2024-09-30T06:27:53Z)
• Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment [8.948475969696075]
  Large Language Models (LLMs) are powerful zero-shot assessors used in real-world situations such as assessing written exams and benchmarking systems. We show that short universal adversarial phrases can be deceived to judge LLMs to predict inflated scores. It is found that judge-LLMs are significantly more susceptible to these adversarial attacks when used for absolute scoring.
  arXiv  Detail & Related papers  (2024-02-21T18:55:20Z)
• Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation [49.3814117521631]
  Standard benchmarks of bias and fairness in large language models (LLMs) measure the association between the user attributes stated or implied by a prompt.<n>We develop analogous RUTEd evaluations from three contexts of real-world use: children's bedtime stories, user personas, and English language learning exercises.<n>We find that standard bias metrics have no significant correlation with the more realistic bias metrics.
  arXiv  Detail & Related papers  (2024-02-20T01:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.