Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation

• URL: http://arxiv.org/abs/2502.10762v2
• Date: Mon, 02 Jun 2025 12:51:19 GMT
• Title: Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation
• Authors: Guofu Xie, Xiao Zhang, Ting Yao, Yunsheng Shi,
• Abstract summary: Bone Soup is a novel model merging approach that first seeks a series of backbone models and then makes the soup (i.e., merge the backbone models)<n>We show that Bone Soup exhibits strong controllability and Pareto optimality in controllable multi-objective generation.
• Score: 42.662194131372125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: User information needs are often highly diverse and varied. A key challenge in current research is how to achieve controllable multi-objective generation while enabling rapid adaptation to accommodate diverse user demands during test time. Existing solutions, such as Rewarded Soup, focus on merging language models individually tuned on single objectives. While easy to implement and widely used, these approaches face limitations in achieving optimal performance due to their disregard for the impacts of competing objectives on model tuning. To address this issue, we propose Bone Soup, a novel model merging approach that first seeks a series of backbone models by considering the impacts of multiple objectives and then makes the soup (i.e., merge the backbone models). Specifically, Bone Soup begins by training multiple backbone models for different objectives using multi-objective reinforcement learning. Each backbone model is guided by a combination of backbone reward signals. To ensure that these models are optimal for the Pareto front, the backbone rewards are crafted by combining standard reward functions into basis vectors, which can then be modified through a rule-based construction method. Bone Soup leverages a symmetric circulant matrix mapping to generate the merging coefficients, which are used to merge the backbone models according to user preferences. Extensive experimental results demonstrate that Bone Soup exhibits strong controllability and Pareto optimality in controllable multi-objective generation, providing a more effective and efficient approach to addressing diverse user needs at test time.

Related papers

• AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling [23.81351558826977]
  We propose AdaJudge, a unified framework that jointly adapts representation and aggregation.<n>AdaJudge first refines backbone representations into a discrimination-oriented space via refinement blocks.<n>It then replaces the static readout with an adaptive multi-view pooling module that dynamically routes and combines evidence.
  arXiv  Detail & Related papers  (2026-01-13T00:37:38Z)
• Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models [14.643457726166632]
  We propose a probing-based adapter that integrates features from multiple models and layers.<n>ComBo does not require dataset-specific tuning or backpropagation through the backbone models.<n>Our results demonstrate that ComBo offers a practical and general-purpose framework for combining diverse representations from multiple FMs.
  arXiv  Detail & Related papers  (2025-12-01T08:26:08Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation [49.98025799046136]
  We introduce Merge-And-GuidE, a two-stage framework that leverages model merging for guided decoding.<n>In Stage 1, MAGE resolves a compatibility problem between the guidance and base models.<n>In Stage 2, we merge explicit and implicit value models into a unified guidance proxy, which then steers the decoding of the base model from Stage 1.
  arXiv  Detail & Related papers  (2025-10-04T11:10:07Z)
• Intention-Conditioned Flow Occupancy Models [69.79049994662591]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• EMORL: Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuning [6.675088737484839]
  We introduce an Ensemble Multi-Objective RL (EMORL) framework that fine-tunes multiple models with individual objectives to improve efficiency and flexibility.<n>Our method is the first to aggregate the hidden states of individual models, incorporating contextual information from multiple objectives.<n>We demonstrate the advantages of EMORL against existing baselines in experiments on the PAIR and Psych8k datasets.
  arXiv  Detail & Related papers  (2025-05-05T11:30:46Z)
• Robust Multi-Objective Preference Alignment with Online DPO [6.434799451791957]
  Multi-objective preference alignment is critical for developing AI systems that are personalizable, helpful, and safe. Existing approaches are either computationally expensive to train or do not sufficiently steer model behaviors. This paper introduces the Multi-Objective Online DPO algorithm, designed to robustly and efficiently align model behaviors with multiple, potentially conflicting human preferences.
  arXiv  Detail & Related papers  (2025-03-01T02:01:49Z)
• CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition [10.045163723630159]
  CHASE operates as a sample-adaptive normalization method to mitigate inter-entity distribution discrepancies.<n>Our approach seamlessly adapts to single-entity backbones and boosts their performance in multi-entity scenarios.
  arXiv  Detail & Related papers  (2024-10-09T17:55:43Z)
• Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Merging [11.186194228460273]
  We propose a preference-aware model merging problem in which the performance of the merged model on each base model's task is treated as an objective. We show that the proposed model merging produces diverse trade-off models and achieves higher test accuracy compared to state-of-the-art merging baselines.
  arXiv  Detail & Related papers  (2024-08-22T03:41:14Z)
• PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences [6.398937923320069]
  We propose PAL, a framework to model human preference complementary to existing pretraining strategies. We show that PAL achieves competitive reward model accuracy compared to strong baselines.
  arXiv  Detail & Related papers  (2024-06-12T17:54:54Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• FissionFusion: Fast Geometric Generation and Hierarchical Souping for Medical Image Analysis [0.7751705157998379]
  The scarcity of well-annotated medical datasets requires leveraging transfer learning from broader datasets like ImageNet or pre-trained models like CLIP. Model soups averages multiple fine-tuned models aiming to improve performance on In-Domain (ID) tasks and enhance robustness against Out-of-Distribution (OOD) datasets. We propose a hierarchical merging approach that involves local and global aggregation of models at various levels.
  arXiv  Detail & Related papers  (2024-03-20T06:48:48Z)
• Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization [76.09576643028362]
  We present Multi-Objective Direct Preference Optimization (MODPO) for multiple alignment objectives. MODPO folds language modeling directly into reward modeling, training language models as implicit collective reward models. It theoretically yields the same optimal solutions as MORLHF but is practically more stable and efficient.
  arXiv  Detail & Related papers  (2023-10-05T17:35:26Z)
• BOtied: Multi-objective Bayesian optimization with tied multivariate ranks [33.414682601242006]
  In this paper, we show a natural connection between non-dominated solutions and the extreme quantile of the joint cumulative distribution function. Motivated by this link, we propose the Pareto-compliant CDF indicator and the associated acquisition function, BOtied. Our experiments on a variety of synthetic and real-world problems demonstrate that BOtied outperforms state-of-the-art MOBO acquisition functions.
  arXiv  Detail & Related papers  (2023-06-01T04:50:06Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning [85.55727213502402]
  We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. We propose Sample-specific Ensemble of Source Models (SESoM) SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
  arXiv  Detail & Related papers  (2022-10-23T01:33:16Z)
• A Twin Neural Model for Uplift [59.38563723706796]
  Uplift is a particular case of conditional treatment effect modeling. We propose a new loss function defined by leveraging a connection with the Bayesian interpretation of the relative risk. We show our proposed method is competitive with the state-of-the-art in simulation setting and on real data from large scale randomized experiments.
  arXiv  Detail & Related papers  (2021-05-11T16:02:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.