Agent Skill Acquisition for Large Language Models via CycleQD

• URL: http://arxiv.org/abs/2410.14735v4
• Date: Mon, 17 Feb 2025 06:26:58 GMT
• Title: Agent Skill Acquisition for Large Language Models via CycleQD
• Abstract summary: CycleQD is a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm.<n>Each task's performance metric is alternated as the quality measure while the others serve as the behavioral characteristics.<n>Applying CycleQD to LLAMA3-8B-INSTRUCT based models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with GPT-3.5-TURBO.
• Score: 2.749898166276853
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance. To address these challenges, we introduce CycleQD, a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm, along with a model merging based crossover and an SVD-based mutation. In CycleQD, each task's performance metric is alternated as the quality measure while the others serve as the behavioral characteristics. This cyclic focus on individual tasks allows for concentrated effort on one task at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function. Empirical results from AgentBench indicate that applying CycleQD to LLAMA3-8B-INSTRUCT based models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with GPT-3.5-TURBO, which potentially contains much more parameters, across these domains. Crucially, this enhanced performance is achieved while retaining robust language capabilities, as evidenced by its performance on widely adopted language benchmark tasks. We highlight the key design choices in CycleQD, detailing how these contribute to its effectiveness. Furthermore, our method is general and can be applied to image segmentation models, highlighting its applicability across different domains.

Related papers

• Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning [4.227963374833712]
  Generalising to unseen tasks remains a fundamental challenge in offline multi-agent reinforcement learning.<n>We show that scaling task diversity is the dominant factor in achieving robust zero-shot transfer.
  arXiv  Detail & Related papers  (2026-09-03T11:04:23Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility [71.76390626651254]
  We propose SPARD, a framework that establishes an automated, self-paced curriculum by perceiving learning progress to adjust dynamically multi-objective reward weights and data importance.<n>Extensive experiments across multiple benchmarks demonstrate that SPARD significantly enhances model capabilities across all domains.
  arXiv  Detail & Related papers  (2026-04-09T05:37:22Z)
• Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance [92.72779885657373]
  We propose a framework that grounds model selection in the internal functional dynamics of the visual encoder.<n>Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment.<n>Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks.
  arXiv  Detail & Related papers  (2026-02-01T17:29:43Z)
• Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models [28.20124264650572]
  Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across tasks.<n>They often exhibit difficulty in distinguishing task-relevant from irrelevant signals, particularly in tasks like Visual Question Answering (VQA)<n>This vulnerability becomes more evident in modality-specific tasks such as image classification or pure text question answering.<n>We propose a novel framework to fine-tune MLLMs, including perturbation-based data augmentation with both perturbations and adversarial perturbations.
  arXiv  Detail & Related papers  (2025-05-26T07:31:32Z)
• Improving Large Language Model Planning with Action Sequence Similarity [50.52049888490524]
  In this work, we explore how to improve the model planning capability through in-context learning (ICL)<n>We propose GRASE-DC: a two-stage pipeline that first re-samples high AS exemplars and then curates the selected exemplars.<n>Our experimental result confirms that GRASE-DC achieves significant performance improvement on various planning tasks.
  arXiv  Detail & Related papers  (2025-05-02T05:16:17Z)
• Large Language Model as Meta-Surrogate for Data-Driven Many-Task Optimization: A Proof-of-Principle Study [11.452011929848844]
  This study proposes a novel meta-surrogate framework to assist many-task optimization. We formulate a unified framework for many-task fitness prediction, by defining a universal model with metadata to fit a group of problems. Our framework supports dual-level knowledge transfer -- at both the surrogate and individual levels -- enhancing optimization efficiency and robustness.
  arXiv  Detail & Related papers  (2025-03-11T11:13:11Z)
• Learning Task Representations from In-Context Learning [73.72066284711462]
  Large language models (LLMs) have demonstrated remarkable proficiency in in-context learning. We introduce an automated formulation for encoding task information in ICL prompts as a function of attention heads. We show that our method's effectiveness stems from aligning the distribution of the last hidden state with that of an optimally performing in-context-learned model.
  arXiv  Detail & Related papers  (2025-02-08T00:16:44Z)
• Revisiting Weight Averaging for Model Merging [16.503826062785773]
  Model merging aims to build a multi-task learner by combining the parameters of individually fine-tuned models without additional training. Weight averaging implicitly induces task vectors centered around the weight averaging itself. Applying a low-rank approximation to these centered task vectors significantly improves merging performance.
  arXiv  Detail & Related papers  (2024-12-11T06:29:20Z)
• Benchmarking General-Purpose In-Context Learning [19.40952728849431]
  In-context learning (ICL) empowers generative models to address new tasks effectively and efficiently on the fly. In this paper, we study extending ICL to address a broader range of tasks with an extended learning horizon and higher improvement potential. We introduce two benchmarks specifically crafted to train and evaluate GPICL functionalities.
  arXiv  Detail & Related papers  (2024-05-27T14:50:42Z)
• A Large-Scale Evaluation of Speech Foundation Models [110.95827399522204]
  We establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the foundation model paradigm for speech. We propose a unified multi-tasking framework to address speech processing tasks in SUPERB using a frozen foundation model followed by task-specialized, lightweight prediction heads.
  arXiv  Detail & Related papers  (2024-04-15T00:03:16Z)
• InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning [17.66308231838553]
  We propose a novel multi-task learning (MTL) architecture designed to mitigate task interference while optimizing inference computational efficiency. We employ a learnable gating mechanism to automatically balance the shared and task-specific representations while preserving the performance of all tasks.
  arXiv  Detail & Related papers  (2024-02-26T18:59:52Z)
• Task-Distributionally Robust Data-Free Meta-Learning [99.56612787882334]
  Data-Free Meta-Learning (DFML) aims to efficiently learn new tasks by leveraging multiple pre-trained models without requiring their original training data. For the first time, we reveal two major challenges hindering their practical deployments: Task-Distribution Shift ( TDS) and Task-Distribution Corruption (TDC)
  arXiv  Detail & Related papers  (2023-11-23T15:46:54Z)
• Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models [80.23791222509644]
  Inconsistent AI models are considered brittle and untrustworthy by human users. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks. We propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets.
  arXiv  Detail & Related papers  (2023-03-28T16:57:12Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)
• Exploring and Predicting Transferability across NLP Tasks [115.6278033699853]
  We study the transferability between 33 NLP tasks across three broad classes of problems. Our results show that transfer learning is more beneficial than previously thought. We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task.
  arXiv  Detail & Related papers  (2020-05-02T09:39:36Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)
• CALM: Continuous Adaptive Learning for Language Modeling [18.72860206714457]
  Training large language representation models has become a standard in the natural language processing community. We demonstrate that in practice these pre-trained models present performance deterioration in the form of catastrophic forgetting. We propose CALM, Continuous Adaptive Learning for Language Modeling: techniques to render models which retain knowledge across multiple domains.
  arXiv  Detail & Related papers  (2020-04-08T03:51:17Z)
• Modelling Latent Skills for Multitask Language Generation [15.126163032403811]
  We present a generative model for multitask conditional language generation. Our guiding hypothesis is that a shared set of latent skills underlies many disparate language generation tasks. We instantiate this task embedding space as a latent variable in a latent variable sequence-to-sequence model.
  arXiv  Detail & Related papers  (2020-02-21T20:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.