Can Models Learn Skill Composition from Examples?

• URL: http://arxiv.org/abs/2409.19808v1
• Date: Sun, 29 Sep 2024 22:14:02 GMT
• Title: Can Models Learn Skill Composition from Examples?
• Authors: Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, Sanjeev Arora,
• Abstract summary: We evaluate the capacity of smaller models to learn compositional generalization from examples. We show that training on combinations of $k=2$ and $3$ skills results in noticeable improvements in the ability to compose texts. This study also suggests that incorporating skill-rich (potentially synthetic) text into training can substantially enhance the compositional capabilities of models.
• Score: 50.5142714905768
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models (LLMs) become increasingly advanced, their ability to exhibit compositional generalization -- the capacity to combine learned skills in novel ways not encountered during training -- has garnered significant attention. This type of generalization, particularly in scenarios beyond training data, is also of great interest in the study of AI safety and alignment. A recent study introduced the SKILL-MIX evaluation, where models are tasked with composing a short paragraph demonstrating the use of a specified $k$-tuple of language skills. While small models struggled with composing even with $k=3$, larger models like GPT-4 performed reasonably well with $k=5$ and $6$. In this paper, we employ a setup akin to SKILL-MIX to evaluate the capacity of smaller models to learn compositional generalization from examples. Utilizing a diverse set of language skills -- including rhetorical, literary, reasoning, theory of mind, and common sense -- GPT-4 was used to generate text samples that exhibit random subsets of $k$ skills. Subsequent fine-tuning of 7B and 13B parameter models on these combined skill texts, for increasing values of $k$, revealed the following findings: (1) Training on combinations of $k=2$ and $3$ skills results in noticeable improvements in the ability to compose texts with $k=4$ and $5$ skills, despite models never having seen such examples during training. (2) When skill categories are split into training and held-out groups, models significantly improve at composing texts with held-out skills during testing despite having only seen training skills during fine-tuning, illustrating the efficacy of the training approach even with previously unseen skills. This study also suggests that incorporating skill-rich (potentially synthetic) text into training can substantially enhance the compositional capabilities of models.

Related papers

• Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models [61.467781476005435]
  skill-wise performance is obscured when inspecting aggregate accuracy, under-utilizing the rich signal modern benchmarks contain. We propose an automatic approach to recover the underlying skills relevant for any evaluation instance, by way of inspecting model-generated rationales. Our skill-slices and framework open a new avenue in model evaluation, leveraging skill-specific analyses to unlock a more granular and actionable understanding of model capabilities.
  arXiv  Detail & Related papers  (2024-10-17T17:51:40Z)
• A Mathematical Theory for Learning Semantic Languages by Abstract Learners [9.139188656944429]
  We develop a mathematical theory to explain the emergence of learned skills, taking the learning process into account. We demonstrate the emergence of learned skills when the ratio of the number of training texts to the number of skills exceeds a certain threshold. We use site percolation analysis to derive the conditions for the existence of a giant component in the skill association graph.
  arXiv  Detail & Related papers  (2024-04-10T13:50:46Z)
• Rethinking Skill Extraction in the Job Market Domain using Large Language Models [20.256353240384133]
  Skill Extraction involves identifying skills and qualifications mentioned in documents such as job postings and resumes. The reliance on manually annotated data limits the generalizability of such approaches. In this paper, we explore the use of in-context learning to overcome these challenges.
  arXiv  Detail & Related papers  (2024-02-06T09:23:26Z)
• Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models [50.11814354654953]
  Key ability of an AI agent is to flexibly combine, as needed, the basic skills it has learned. This work introduces Skill-Mix, a new evaluation to measure ability to combine skills.
  arXiv  Detail & Related papers  (2023-10-26T16:55:05Z)
• UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks [60.46473247205654]
  Using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models.
  arXiv  Detail & Related papers  (2023-06-07T18:26:22Z)
• Skill-Based Few-Shot Selection for In-Context Learning [123.26522773708683]
  Skill-KNN is a skill-based few-shot selection method for in-context learning. It does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks. Experimental results across five cross-domain semantic parsing datasets and six backbone models show that Skill-KNN significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2023-05-23T16:28:29Z)
• Pre-Training to Learn in Context [138.0745138788142]
  The ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. We propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters.
  arXiv  Detail & Related papers  (2023-05-16T03:38:06Z)
• CREPE: Can Vision-Language Foundation Models Reason Compositionally? [10.958279688917434]
  We introduce a new compositionality evaluation benchmark, CREPE, which measures two important aspects of compositionality identified by cognitive science literature: systematicity and productivity. For systematicity, we find that model performance decreases consistently when novel compositions dominate the retrieval set. For productivity, models' retrieval success decays as complexity increases, frequently nearing random chance at high complexity.
  arXiv  Detail & Related papers  (2022-12-13T19:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.