ProFuser: Progressive Fusion of Large Language Models

• URL: http://arxiv.org/abs/2408.04998v1
• Date: Fri, 9 Aug 2024 11:18:29 GMT
• Title: ProFuser: Progressive Fusion of Large Language Models
• Authors: Tianyuan Shi, Fanqi Wan, Canbin Huang, Xiaojun Quan, Chenliang Li, Ming Yan, Ji Zhang,
• Abstract summary: We introduce a novel approach that enhances the fusion process by incorporating both the training and inference modes. Our method evaluates model advantage not only through cross entropy during training but also by considering inference outputs. To validate ProFuser's effectiveness, we fused three models, including vicuna-7b-v1.5, Llama-2-7b-chat, and mpt-7b-8k-chat.
• Score: 53.697927989207045
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While fusing the capacities and advantages of various large language models (LLMs) offers a pathway to construct more powerful and versatile models, a fundamental challenge is to properly select advantageous model during the training. Existing fusion methods primarily focus on the training mode that uses cross entropy on ground truth in a teacher-forcing setup to measure a model's advantage, which may provide limited insight towards model advantage. In this paper, we introduce a novel approach that enhances the fusion process by incorporating both the training and inference modes. Our method evaluates model advantage not only through cross entropy during training but also by considering inference outputs, providing a more comprehensive assessment. To combine the two modes effectively, we introduce ProFuser to progressively transition from inference mode to training mode. To validate ProFuser's effectiveness, we fused three models, including vicuna-7b-v1.5, Llama-2-7b-chat, and mpt-7b-8k-chat, and demonstrated the improved performance in knowledge, reasoning, and safety compared to baseline methods.

Related papers

• SNOOPI: Supercharged One-step Diffusion Distillation with Proper Guidance [12.973835034100428]
  This paper presents SNOOPI, a novel framework designed to enhance the guidance in one-step diffusion models during both training and inference. By varying the guidance scale of both teacher models, we broaden their output distributions, resulting in a more robust VSD loss that enables SB to perform effectively across diverse backbones while maintaining competitive performance. Second, we propose a training-free method called Negative-Away Steer Attention (NASA), which integrates negative prompts into one-step diffusion models via cross-attention to suppress undesired elements in generated images.
  arXiv  Detail & Related papers  (2024-12-03T18:56:32Z)
• Joint Training for Selective Prediction [5.662924503089369]
  Selective Prediction methods determine when to adopt a classifier's output versus defer to a human. One previous method involves learning a deferral model based on engineered features. We introduce a novel joint-training approach that simultaneously optimize learned representations used by the classifier module and a learned deferral policy.
  arXiv  Detail & Related papers  (2024-10-31T15:28:26Z)
• Training-free Diffusion Model Alignment with Sampling Demons [15.400553977713914]
  We propose an optimization approach, dubbed Demon, to guide the denoising process at inference time without backpropagation through reward functions or model retraining. Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through optimization. To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models.
  arXiv  Detail & Related papers  (2024-10-08T07:33:49Z)
• CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models [77.39903417768967]
  CatVTON is a virtual try-on diffusion model that transfers in-shop or worn garments of arbitrary categories to target individuals. CatVTON consists only of a VAE and a simplified denoising UNet, removing redundant image and text encoders. Experiments demonstrate that CatVTON achieves superior qualitative and quantitative results compared to baseline methods.
  arXiv  Detail & Related papers  (2024-07-21T11:58:53Z)
• Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment [65.15914284008973]
  We propose to leverage an Inverse Reinforcement Learning (IRL) technique to simultaneously build an reward model and a policy model. We show that the proposed algorithms converge to the stationary solutions of the IRL problem. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process.
  arXiv  Detail & Related papers  (2024-05-28T07:11:05Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)
• Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models [52.1809084559048]
  We propose a novel two-stage divide-and-conquer training strategy termed TDC Training. It groups timesteps based on task similarity and difficulty, assigning highly customized denoising models to each group, thereby enhancing the performance of diffusion models. While two-stage training avoids the need to train each model separately, the total training cost is even lower than training a single unified denoising model.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval [3.164991885881342]
  Cross- retrieval models learn robust embedding spaces. We introduce a novel approach rooted in curriculum learning to address this problem. We propose a two-stage training paradigm that guides the model's learning process from semi-hard to hard triplets.
  arXiv  Detail & Related papers  (2023-10-20T12:35:54Z)
• Efficient Multimodal Fusion via Interactive Prompting [62.08292938484994]
  Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era. We propose an efficient and flexible multimodal fusion method, namely PMF, tailored for fusing unimodally pre-trained transformers.
  arXiv  Detail & Related papers  (2023-04-13T07:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.