Improving Weak-to-Strong Generalization with Scalable Oversight and
Ensemble Learning
- URL: http://arxiv.org/abs/2402.00667v1
- Date: Thu, 1 Feb 2024 15:30:19 GMT
- Title: Improving Weak-to-Strong Generalization with Scalable Oversight and
Ensemble Learning
- Authors: Jitao Sang, Yuhang Wang, Jing Zhang, Yanxu Zhu, Chao Kong, Junhong Ye,
Shuyu Wei and Jinlin Xiao
- Abstract summary: This paper presents a follow-up study to OpenAI's recent superalignment work on Weak-to-Strong Generalization (W2SG)
Superalignment focuses on ensuring that high-level AI systems remain consistent with human values and intentions when dealing with complex, high-risk tasks.
Our study simulates two phases of superalignment under the W2SG framework: the development of general superhuman models and the progression towards superintelligence.
- Score: 21.401598876308345
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper presents a follow-up study to OpenAI's recent superalignment work
on Weak-to-Strong Generalization (W2SG). Superalignment focuses on ensuring
that high-level AI systems remain consistent with human values and intentions
when dealing with complex, high-risk tasks. The W2SG framework has opened new
possibilities for empirical research in this evolving field. Our study
simulates two phases of superalignment under the W2SG framework: the
development of general superhuman models and the progression towards
superintelligence. In the first phase, based on human supervision, the quality
of weak supervision is enhanced through a combination of scalable oversight and
ensemble learning, reducing the capability gap between weak teachers and strong
students. In the second phase, an automatic alignment evaluator is employed as
the weak supervisor. By recursively updating this auto aligner, the
capabilities of the weak teacher models are synchronously enhanced, achieving
weak-to-strong supervision over stronger student models.We also provide an
initial validation of the proposed approach for the first phase. Using the SciQ
task as example, we explore ensemble learning for weak teacher models through
bagging and boosting. Scalable oversight is explored through two auxiliary
settings: human-AI interaction and AI-AI debate. Additionally, the paper
discusses the impact of improved weak supervision on enhancing weak-to-strong
generalization based on in-context learning. Experiment code and dataset will
be released at https://github.com/ADaM-BJTU/W2SG.
Related papers
- Bayesian WeakS-to-Strong from Text Classification to Generation [14.897191979004782]
This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions.
Confidence scores are estimated using a Bayesian approach to guide the WeakS-to-Strong generalization.
Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superalignment.
arXiv Detail & Related papers (2024-05-24T13:33:11Z) - Co-Supervised Learning: Improving Weak-to-Strong Generalization with
Hierarchical Mixture of Experts [81.37287967870589]
We propose to harness a diverse set of specialized teachers, instead of a single generalist one, that collectively supervises the strong student.
Our approach resembles the classical hierarchical mixture of experts, with two components tailored for co-supervision.
We validate the proposed method through visual recognition tasks on the OpenAI weak-to-strong benchmark and additional multi-domain datasets.
arXiv Detail & Related papers (2024-02-23T18:56:11Z) - Vision Superalignment: Weak-to-Strong Generalization for Vision
Foundation Models [55.919653720979824]
This paper focuses on the concept of weak-to-strong generalization, which involves using a weaker model to supervise a stronger one.
We introduce a novel and adaptively adjustable loss function for weak-to-strong supervision.
Our approach not only exceeds the performance benchmarks set by strong-to-strong generalization but also surpasses the outcomes of fine-tuning strong models with whole datasets.
arXiv Detail & Related papers (2024-02-06T06:30:34Z) - A General Framework for Learning from Weak Supervision [93.89870459388185]
This paper introduces a general framework for learning from weak supervision (GLWS) with a novel algorithm.
Central to GLWS is an Expectation-Maximization (EM) formulation, adeptly accommodating various weak supervision sources.
We also present an advanced algorithm that significantly simplifies the EM computational demands.
arXiv Detail & Related papers (2024-02-02T21:48:50Z) - Artificial Intelligence and Dual Contract [2.1756081703276]
We develop a model where two principals, each equipped with independent Q-learning algorithms, interact with a single agent.
Our findings reveal that the strategic behavior of AI principals hinges crucially on the alignment of their profits.
arXiv Detail & Related papers (2023-03-22T07:31:44Z) - TWINS: A Fine-Tuning Framework for Improved Transferability of
Adversarial Robustness and Generalization [89.54947228958494]
This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks.
We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework.
TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
arXiv Detail & Related papers (2023-03-20T14:12:55Z) - Learning Dynamics and Generalization in Reinforcement Learning [59.530058000689884]
We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training.
We show that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly networks and gradient networks trained with policy methods.
arXiv Detail & Related papers (2022-06-05T08:49:16Z) - Alleviating Robust Overfitting of Adversarial Training With Consistency
Regularization [9.686724616328874]
Adversarial training (AT) has proven to be one of the most effective ways to defend Deep Neural Networks (DNNs) against adversarial attacks.
robustness will drop sharply at a certain stage, always exists during AT.
consistency regularization, a popular technique in semi-supervised learning, has a similar goal as AT and can be used to alleviate robust overfitting.
arXiv Detail & Related papers (2022-05-24T03:18:43Z) - Weakly Supervised Semantic Segmentation via Alternative Self-Dual
Teaching [82.71578668091914]
This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model.
We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
arXiv Detail & Related papers (2021-12-17T11:56:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.