Contrastive Weak-to-strong Generalization

• URL: http://arxiv.org/abs/2510.07884v1
• Date: Thu, 09 Oct 2025 07:37:23 GMT
• Title: Contrastive Weak-to-strong Generalization
• Authors: Houcheng Jiang, Junfeng Fang, Jiaxin Wu, Tianyu Zhang, Chen Gao, Yong Li, Xiang Wang, Xiangnan He, Yang Deng,
• Abstract summary: We propose Contrastive Weak-to-Strong Generalization (ConG) to advance weak-to-strong generalization.<n>This framework employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples.
• Score: 50.5986177336082
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose Contrastive Weak-to-Strong Generalization (ConG), a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.

Related papers

• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling [49.41422138354821]
  We propose a principled reward modeling framework that integrates non-negative factor analysis into the Bradley-Terry preference model.<n>BNRM represents rewards through a sparse, non-negative latent factor generative process.<n>We show that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.
  arXiv  Detail & Related papers  (2026-02-11T08:14:11Z)
• Proxy Robustness in Vision Language Models is Effortlessly Transferable [13.390016978827163]
  A pivotal technique for improving the defense of deep models, adversarial robustness transfer via distillation has demonstrated remarkable success in conventional image classification tasks.<n>We bridge this gap by revealing an interesting phenomenon: vanilla CLIP (without adversarial training) exhibits intrinsic defensive capabilities against adversarial examples.<n>We formally define this as proxy adversarial robustness, and naturally propose a Heterogeneous Proxy Transfer framework.
  arXiv  Detail & Related papers  (2026-01-19T09:23:11Z)
• Weak-to-Strong Generalization under Distribution Shifts [6.711930932187631]
  We propose RAVEN, a robust weak-to-strong generalization framework.<n>RAVEN learns the optimal combinations of weak models in addition to parameters of the strong model.<n>Our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.
  arXiv  Detail & Related papers  (2025-10-24T10:46:50Z)
• Harnessing Consistency for Robust Test-Time LLM Ensemble [88.55393815158608]
  CoRE is a plug-and-play technique that harnesses model consistency for robust LLM ensemble.<n> Token-level consistency captures fine-grained disagreements by applying a low-pass filter to downweight uncertain tokens.<n>Model-level consistency models global agreement by promoting model outputs with high self-confidence.
  arXiv  Detail & Related papers  (2025-10-12T04:18:45Z)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models [62.70575022567081]
  We propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning.<n>Our work establishes a new direction for building more robust and reliable reasoning models.
  arXiv  Detail & Related papers  (2025-09-29T04:27:23Z)
• When Shift Happens - Confounding Is to Blame [21.028845817427825]
  We show that empirical risk minimization can rival or even outperform state-of-the-art out-of-distribution generalization methods.<n>We attribute this phenomenon to hidden confounding.<n>We show that models augmented with proxies for hidden confounders can mitigate the challenges posed by hidden confounding shifts.
  arXiv  Detail & Related papers  (2025-05-27T16:50:44Z)
• Unified Enhancement of the Generalization and Robustness of Language Models via Bi-Stage Optimization [2.502393972789905]
  We propose a bi-stage optimization framework to uniformly enhance both the generalization and robustness of LMs.<n>We show that our method significantly improves the generalization and robustness of LMs compared to other existing methods.
  arXiv  Detail & Related papers  (2025-03-19T13:50:36Z)
• Enhancing NLP Robustness and Generalization through LLM-Generated Contrast Sets: A Scalable Framework for Systematic Evaluation and Adversarial Training [0.0]
  We create a 3,000-example contrast set to evaluate and improve model robustness.<n>Fine-tuning on these contrast sets enhanced performance on systematically perturbed examples, maintained standard test accuracy, and modestly improved generalization to novel perturbations.
  arXiv  Detail & Related papers  (2025-03-09T14:52:53Z)
• How to Mitigate Overfitting in Weak-to-strong Generalization? [50.37526669608372]
  Weak-to-strong generalization aims to elicit the capabilities of strong models through weak supervisors.<n>Strong models exhibit significant overfitting in weak-to-strong generalization.<n>We propose a two-stage framework that simultaneously improves the quality of supervision signals and the quality of input questions.
  arXiv  Detail & Related papers  (2025-03-06T09:32:39Z)
• Transferable Adversarial Attacks on SAM and Its Downstream Models [87.23908485521439]
  This paper explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM)<n>To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal meta-initialization (UMI) algorithm.
  arXiv  Detail & Related papers  (2024-10-26T15:04:04Z)
• Revisiting the Robust Generalization of Adversarial Prompt Tuning [4.033827046965844]
  We propose an adaptive Consistency-guided Adrial Prompt Tuning (i.e., CAPT) framework to enhance the alignment of image and text features for adversarial examples. We conduct experiments across 14 datasets and 4 data sparsity schemes to show the superiority of CAPT over other state-of-the-art adaption methods.
  arXiv  Detail & Related papers  (2024-05-18T02:54:41Z)
• Generating Less Certain Adversarial Examples Improves Robust Generalization [22.00283527210342]
  This paper revisits the robust overfitting phenomenon of adversarial training.<n>We argue that overconfidence in predicting adversarial examples is a potential cause.<n>We propose a formal definition of adversarial certainty that captures the variance of the model's predicted logits on adversarial examples.
  arXiv  Detail & Related papers  (2023-10-06T19:06:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.