The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards

• URL: http://arxiv.org/abs/2609.13866v1
• Date: Sat, 12 Sep 2026 10:36:55 GMT
• Title: The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards
• Abstract summary: Group-relative policy optimization can discard nocontrast rollout groups through dynamic sampling.<n>We identify and quantify a metricpredicate mismatch under composite shaped rewards.
• Score: 1.667501390377158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Group-relative policy optimization (GRPO and descendants) can discard no-contrast rollout groups through dynamic sampling, while practical implementations expose a configurable filter metric. We identify and quantify a metric-predicate mismatch under composite shaped rewards. When filtering follows the shaped training score rather than the task outcome, all-fail groups retain nonzero within-group spread and pass the predicate; standard-deviation normalization then promotes shaping differences among failures to full-size phantom advantages. In a controlled GSM8K comparison (Qwen2.5-1.5B, LoRA), no filtering and shaped-score filtering end at EM 0.080 +/- 0.112 and 0.040 +/- 0.008, whereas binary-outcome filtering holds 0.754 +/- 0.005 across four runs per arm (three default-seed reruns and one seed-123 run; mean +/- sample SD). On verl's native recipe/dapo trainer, holding model, data, reward and trainer fixed and changing only the metric, the score arm requires no batch refill in any of 40 observed steps and ends at EM 0.160; the accuracy arm refills in 29/40 steps and ends at 0.763. Both use the same custom shaped-reward hook and unmodified trainer/filter code. Prior work established shaping-induced amplification and all-fail filtering; our contribution isolates the metric-predicate semantic mismatch and directly instruments native deletion/refill telemetry. Across tested positive coefficients lambda in {0.1, 0.3, 0.5}, unsafe arms collapse; exploratory one-run cells reproduce the failure at 1.5B/7B on MATH and under GSPO, while disabling standard-deviation normalization avoids the observed collapse. Filtering under a composite reward should use a task-outcome signal whose semantics are independent of shaping.

Related papers

• WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training [48.290951447286744]
  WDL-OPD is a mixture-constrained co-training method with two trainable policies.<n>An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL.<n>We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.
  arXiv  Detail & Related papers  (2026-08-10T11:22:53Z)
• Confidence Scores in Open-Vocabulary Detection Are a Biased Mixture of Scale and Semantics [8.465978346858842]
  Foundation models such as CLIP have enabled open-vocabulary object detectors that generalise to novel categories via vision-language similarity.<n>We show that the scale-bias finding is a biased mixture of two effects.<n>These findings reveal a fundamental limitation of adapting image-level foundation models to region-level detection tasks.
  arXiv  Detail & Related papers  (2026-07-13T01:33:18Z)
• Underwater Dead Reckoning with Deployable Situation-Triggered Covariance Scheduling [3.2548794659022398]
  This paper presents the Situation-Triggered Calibrated Adaptive Robust Extended Kalman Filter for a BlueROV2.<n>An onboard probabilistic trigger identifies the current motion situation while one error-state filter runs continuously.<n>When the trigger is confident, the filter changes only to the corresponding pre-calibrated process- and measurement-noise matrices.
  arXiv  Detail & Related papers  (2026-07-12T06:27:57Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Reframing preprocessing selection as model-internal calibration in near-infrared spectroscopy: A large-scale benchmark of operator-adaptive PLS and Ridge models [0.2609784101826761]
  This paper studies the case where that search can be collapsed into one calibration step.<n> Linear operator-adaptive calibration gives comparable prediction quality to exhaustive preprocessing screening.
  arXiv  Detail & Related papers  (2026-05-13T14:23:00Z)
• HBEE: Human Behavioral Entropy Engine -- Pre-Registered Multi-Agent LLM Simulation of Peer-Suspicion-Based Detection Inversion [0.0]
  The paper makes a narrow but surprising claim: in a controlled environment where adaptive OPSEC is implementable as an LLM directive, peer-suspicion-cascade detection inverts.<n>We release the simulator, pre-registration document, frozen scenarios, raw telemetry, and analysis pipeline under an open-source license.
  arXiv  Detail & Related papers  (2026-05-08T09:19:21Z)
• Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B [0.0]
  Small instruct-tuned LLMs produce degenerate verbal confidence under minimal elicitation.<n>We test whether confidence-conditioned supervised fine-tuning with self-consistency-derived targets can close the gap between internal information and verbal readout.
  arXiv  Detail & Related papers  (2026-04-27T05:53:26Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• When Domain Pretraining Interferes with Instruction Alignment: An Empirical Study of Adapter Merging in Medical LLMs [0.6345523830122167]
  Large language models can exhibit surprising adapter interference when combining domain adaptation and instruction alignment.<n>We study a two-stage LoRA pipeline for medical LLMs, where domain-oriented pre-training (PT) and supervised fine-tuning (SFT) are trained separately and later merged.
  arXiv  Detail & Related papers  (2026-01-26T10:54:06Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion [139.71852076031962]
  We present a novel filter pruning method, dubbed dynamic-coded filter fusion (DCFF) We derive compact CNNs in a computation-economical and regularization-free manner for efficient image classification. Our DCFF derives a compact VGGNet-16 with only 72.77M FLOPs and 1.06M parameters while reaching top-1 accuracy of 93.47%.
  arXiv  Detail & Related papers  (2021-07-14T18:07:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.