Claim-Level Rubric Rewards for Video Caption Reinforcement Learning

• URL: http://arxiv.org/abs/2607.05150v1
• Date: Mon, 06 Jul 2026 14:33:41 GMT
• Title: Claim-Level Rubric Rewards for Video Caption Reinforcement Learning
• Abstract summary: Claim-Level Rewards Rewards (CuRe) is a structured reward framework designed to address the reward-design bottleneck in reinforcement learning for dense video captioning.<n>CuRe reformulates reward modeling as fine-grained claim-level verification.
• Score: 44.96572145476932
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce Claim-Level Rubric Rewards (CuRe), a structured reward framework designed to address the reward-design bottleneck in reinforcement learning for dense video captioning. Existing reward designs generally fall into two categories: holistic response-level judgment across heterogeneous criteria, or alignment-based evaluation against reference captions. However, both paradigms suffer from fundamental limitations. Holistic rewards struggle to ensure factual accuracy and are prone to stylistic reward hacking, while reference-based rewards overly rely on rigid textual alignment, failing to preserve the completeness and diversity inherent to open-ended generation tasks. To address these challenges, CuRe reformulates reward modeling as fine-grained claim-level verification. Specifically, CuRe decomposes captions into category-aware atomic claims through a structured rubric, converting holistic evaluation into simpler and more reliable claim-level verification.

Related papers

• CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization [21.755134237801443]
  CoRT is a token-level credit weighting method for rubric-conditioned GRPO.<n>We show that CoRT improves over matched response-level GRPO in the vast majority of comparisons.
  arXiv  Detail & Related papers  (2026-07-28T12:45:19Z)
• Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search [8.82040111147814]
  We present an end-to-end RLAIF framework to generate emphportable search queries.<n>This task introduces a highly adversarial reward surface where policy optimization exploits flaws in rubrics, resulting in verbatim-copying behaviors.<n>We show that introducing a rule-based reward floor to correct for rewards assigned to verbatim copying degenerates this failure mode, resulting in a substantial $+0.147$ quality improvement on a cross-family evaluation judge.
  arXiv  Detail & Related papers  (2026-06-25T17:09:12Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation [51.22170603236523]
  We propose a Balanced Token-level Policy Optimization framework for hallucination mitigation.<n>BalTO extracts checkable factual claims, verifies them against the reference context, and projects claim-level judgments to token-level labels.<n>Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show that BALTO achieves the highest faithfulness across all six model--benchmark settings.
  arXiv  Detail & Related papers  (2026-06-14T16:25:59Z)
• Prompt-Level Reward Specifications for Open-Ended Post-Training [43.215398565987265]
  Open-ended post-training benefits from rewards that make prompt-specific success conditions explicit.<n>We propose a prompt-level reward specification framework that separates reward specification from reward.
  arXiv  Detail & Related papers  (2026-05-28T02:52:06Z)
• Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning [52.48243762705385]
  Chain of Uncertain Rewards (CoUR) is a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation.<n>We show that CoUR achieves better performance and significantly lowers the cost of reward evaluations.
  arXiv  Detail & Related papers  (2026-04-15T05:44:14Z)
• Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks [30.108876878365496]
  Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models with complex, open-domain instruction following tasks.<n>Existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems.<n>We propose a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment.
  arXiv  Detail & Related papers  (2026-04-03T07:02:57Z)
• Reinforcing Consistency in Video MLLMs with Structured Rewards [14.560061824569333]
  Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding.<n>We study this failure mode through a compositional consistency audit that decomposes a caption into supporting factual and temporal claims.<n>Our training objective integrates three complementary components: (1) an instance-aware scene-graph reward for factual objects, attributes, and relations; (2) a temporal reward for event ordering and repetition; and (3) a video-grounded VQA reward for hierarchical self-verification.
  arXiv  Detail & Related papers  (2026-04-01T23:15:34Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA [46.65999744568314]
  We introduce QA-LIGN, which decomposes monolithic rewards into interpretable principle-specific evaluations.<n>Applying to uncensored Llama-3.1-8B-Instruct, QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate.
  arXiv  Detail & Related papers  (2025-06-09T18:24:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.