OpenICL: An Open-Source Framework for In-context Learning
- URL: http://arxiv.org/abs/2303.02913v1
- Date: Mon, 6 Mar 2023 06:20:25 GMT
- Title: OpenICL: An Open-Source Framework for In-context Learning
- Authors: Zhenyu Wu, YaoXiang Wang, Jiacheng Ye, Jiangtao Feng, Jingjing Xu, Yu
Qiao, Zhiyong Wu
- Abstract summary: We introduce OpenICL, an open-source toolkit for In-context Learning (ICL) and large language model evaluation.
OpenICL is research-friendly with a highly flexible architecture that users can easily combine different components to suit their needs.
The effectiveness of OpenICL has been validated on a wide range of NLP tasks, including classification, QA, machine translation, and semantic parsing.
- Score: 48.75452105457122
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In recent years, In-context Learning (ICL) has gained increasing attention
and emerged as the new paradigm for large language model (LLM) evaluation.
Unlike traditional fine-tuning methods, ICL instead adapts the pre-trained
models to unseen tasks without any parameter updates. However, the
implementation of ICL is sophisticated due to the diverse retrieval and
inference methods involved, as well as the varying pre-processing requirements
for different models, datasets, and tasks. A unified and flexible framework for
ICL is urgently needed to ease the implementation of the aforementioned
components. To facilitate ICL research, we introduce OpenICL, an open-source
toolkit for ICL and LLM evaluation. OpenICL is research-friendly with a highly
flexible architecture that users can easily combine different components to
suit their needs. It also provides various state-of-the-art retrieval and
inference methods to streamline the process of adapting ICL to cutting-edge
research. The effectiveness of OpenICL has been validated on a wide range of
NLP tasks, including classification, QA, machine translation, and semantic
parsing. As a side-product, we found OpenICL to be an efficient yet robust tool
for LLMs evaluation. OpenICL is released at
https://github.com/Shark-NLP/OpenICL
Related papers
- ICLEval: Evaluating In-Context Learning Ability of Large Language Models [68.7494310749199]
In-Context Learning (ICL) is a critical capability of Large Language Models (LLMs) as it empowers them to comprehend and reason across interconnected inputs.
Existing evaluation frameworks primarily focus on language abilities and knowledge, often overlooking the assessment of ICL ability.
We introduce the ICLEval benchmark to evaluate the ICL abilities of LLMs, which encompasses two key sub-abilities: exact copying and rule learning.
arXiv Detail & Related papers (2024-06-21T08:06:10Z) - Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond [24.151927600694066]
Commit Message Generation (CMG) approaches aim to automatically generate commit messages based on given code diffs.
This paper conducts the first comprehensive experiment to investigate how far we have been in applying Large Language Models (LLMs) to generate high-quality commit messages.
arXiv Detail & Related papers (2024-04-23T08:24:43Z) - LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges.
Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model.
This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
arXiv Detail & Related papers (2024-02-26T07:33:05Z) - RecDCL: Dual Contrastive Learning for Recommendation [65.6236784430981]
We propose a dual contrastive learning recommendation framework -- RecDCL.
In RecDCL, the FCL objective is designed to eliminate redundant solutions on user-item positive pairs.
The BCL objective is utilized to generate contrastive embeddings on output vectors for enhancing the robustness of the representations.
arXiv Detail & Related papers (2024-01-28T11:51:09Z) - Flexibly Scaling Large Language Models Contexts Through Extensible
Tokenization [6.9004592877749005]
Large language models (LLMs) are in need of sufficient contexts to handle many critical applications.
Although the size of context window can be extended by fine-tuning, it will result in a substantial cost in both training and inference stage.
We present Extensible Tokenization as an alternative method which realizes the flexible scaling of LLMs' context.
arXiv Detail & Related papers (2024-01-15T16:00:50Z) - Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning [27.729189318779603]
Batch-ICL is an effective, efficient, and order-agnostic inference algorithm for in-context learning.
We show that Batch-ICL consistently outperforms most permutations of ICL examples.
We also develop a novel variant of Batch-ICL featuring multiple "epochs" of meta-optimization.
arXiv Detail & Related papers (2024-01-12T09:31:17Z) - In-Context Exemplars as Clues to Retrieving from Large Associative
Memory [1.2952137350423816]
In-context learning (ICL) enables large language models (LLMs) to learn patterns from in-context exemplars without training.
How to choose exemplars remains unclear due to the lack of understanding of how in-context learning works.
Our study sheds new light on the mechanism of ICL by connecting it to memory retrieval.
arXiv Detail & Related papers (2023-11-06T20:13:29Z) - FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large
Language Models in Federated Learning [70.38817963253034]
This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution.
We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios.
We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
arXiv Detail & Related papers (2023-09-01T09:40:36Z) - A Survey on In-context Learning [75.41718234460895]
In-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP)
We first present a formal definition of ICL and clarify its correlation to related studies.
We then organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis.
arXiv Detail & Related papers (2022-12-31T15:57:09Z) - Self-Adaptive In-Context Learning: An Information Compression
Perspective for In-Context Example Selection and Ordering [15.3566963926257]
This paper advocates a new principle for in-context learning (ICL): self-adaptive in-context learning.
The self-adaption mechanism is introduced to help each sample find an in-context example permutation that can derive the correct prediction.
Our self-adaptive ICL method achieves a 40% relative improvement over the common practice setting.
arXiv Detail & Related papers (2022-12-20T15:55:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.