The Reciprocity Gradient

• URL: http://arxiv.org/abs/2605.08323v1
• Date: Fri, 08 May 2026 16:29:52 GMT
• Title: The Reciprocity Gradient
• Abstract summary: Communication is fundamental to sustaining reciprocity and cooperation in strategic interactions.<n>We identify and formulate the influence attribution problem as the central optimization difficulty inherent in such dynamics for a learning agent.<n>To address this, we introduce the reciprocity gradient, which explicitly backpropagates reward gradients through private estimators of opponents' policies trained from public observations.
• Score: 52.35929743862925
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Communication is fundamental to sustaining reciprocity and cooperation in strategic interactions. We identify and formulate the influence attribution problem as the central optimization difficulty inherent in such dynamics for a learning agent: any action or signal the agent emits reshapes the reputations of many third parties along combinatorially branching paths before feeding back into its own future rewards, forcing the agent to account for all of these indirect channels at once when choosing every action. To address this, we introduce the reciprocity gradient, which explicitly backpropagates reward gradients through private estimators of opponents' policies trained from public observations. The gradient flows through the reputation chain itself analytically, rather than being estimated from sampled returns. It jointly optimizes actions and evaluative signals without intrinsic rewards or reward shaping. Empirically, the method recovers near-optimal context-sensitive policies, while sample-based baselines collapse into constant-output policies.

Related papers

• Co-Evolving LLM Evaluators and Policies via DynamicRubric [67.2962847914162]
  Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models.<n>We propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework.<n>In experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines.
  arXiv  Detail & Related papers  (2026-07-22T12:35:40Z)
• LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning [12.013691798765594]
  LOTAPO is a self-generated process-supervision method based on backward leave-one-turn attribution.<n>For each search turn, LOTAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder.<n>This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions.
  arXiv  Detail & Related papers  (2026-07-15T06:55:28Z)
• Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target [42.681980014826536]
  Generative recommenders (LLM-Rec) require continual post-deployment updates.<n> deployment logs provide only policy-shaped contextual bandit feedback.<n>We propose an Anchored Bandit Policy Optimization framework for continual LLM-Rec updates.
  arXiv  Detail & Related papers  (2026-05-17T11:10:44Z)
• How RLHF Amplifies Sycophancy [23.213056717401418]
  Large language models often exhibit increased sycophantic behavior after preference-based post-training.<n>We identify an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment.<n>We propose a training-time intervention designed to neutralize the amplification mechanism itself.
  arXiv  Detail & Related papers  (2026-02-01T03:46:14Z)
• Q-learning with Adjoint Matching [58.78551025170267]
  We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm.<n>QAM sidesteps two challenges by leveraging adjoint matching, a recently proposed technique in generative modeling.<n>It consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.
  arXiv  Detail & Related papers  (2026-01-20T18:45:34Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Off-Policy Evaluation for Sequential Persuasion Process with Unobserved Confounding [2.7282382992043885]
  Real-world scenarios often involve hidden variables that impact the receiver's belief formation and decision-making.<n>We conceptualize this as a sequential decision-making problem, where the sender and receiver interact over multiple rounds.<n>By reformulating this scenario as a Partially Observable Markov Decision Process (POMDP), we capture the sender's incomplete information regarding both the dynamics of the receiver's beliefs and the unobserved confounder.
  arXiv  Detail & Related papers  (2025-04-01T21:50:32Z)
• Coherent Soft Imitation Learning [17.345411907902932]
  Imitation learning methods seek to learn from an expert either through behavioral cloning (BC) of the policy or inverse reinforcement learning (IRL) of the reward. This work derives an imitation method that captures the strengths of both BC and IRL.
  arXiv  Detail & Related papers  (2023-05-25T21:54:22Z)
• Where is the Grass Greener? Revisiting Generalized Policy Iteration for Offline Reinforcement Learning [81.15016852963676]
  We re-implement state-of-the-art baselines in the offline RL regime under a fair, unified, and highly factorized framework. We show that when a given baseline outperforms its competing counterparts on one end of the spectrum, it never does on the other end.
  arXiv  Detail & Related papers  (2021-07-03T11:00:56Z)
• Reinforcement Learning with Trajectory Feedback [76.94405309609552]
  In this work, we take a first step towards relaxing this assumption and require a weaker form of feedback, which we refer to as emphtrajectory feedback. Instead of observing the reward obtained after every action, we assume we only receive a score that represents the quality of the whole trajectory observed by the agent, namely, the sum of all rewards obtained over this trajectory. We extend reinforcement learning algorithms to this setting, based on least-squares estimation of the unknown reward, for both the known and unknown transition model cases, and study the performance of these algorithms by analyzing their regret.
  arXiv  Detail & Related papers  (2020-08-13T17:49:18Z)
• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions [18.90946044396516]
  Music streaming, video streaming, news recommendation, and e-commerce services often engage with content in a sequential manner. Providing and evaluating good sequences of recommendations is therefore a central problem for these services. We propose a new counterfactual estimator that allows for sequential interactions in the rewards with lower variance in anally unbiased manner.
  arXiv  Detail & Related papers  (2020-07-25T17:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.