GECKO: Generative Language Model for English, Code and Korean

• URL: http://arxiv.org/abs/2405.15640v1
• Date: Fri, 24 May 2024 15:30:41 GMT
• Title: GECKO: Generative Language Model for English, Code and Korean
• Authors: Sungwoo Oh, Donggyu Kim,
• Abstract summary: We introduce GECKO, a bilingual large language model (LLM) optimized for Korean and English, along with programming languages. GECKO is pretrained on the balanced, high-quality corpus of Korean and English employing LLaMA architecture.
• Score: 0.02046223849354785
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce GECKO, a bilingual large language model (LLM) optimized for Korean and English, along with programming languages. GECKO is pretrained on the balanced, high-quality corpus of Korean and English employing LLaMA architecture. In this report, we share the experiences of several efforts to build a better data pipeline for the corpus and to train our model. GECKO shows great efficiency in token generations for both Korean and English, despite its small size of vocabulary. We measure the performance on the representative benchmarks in terms of Korean, English and Code, and it exhibits great performance on KMMLU (Korean MMLU) and modest performance in English and Code, even with its smaller number of trained tokens compared to English-focused LLMs. GECKO is available to the open-source community under a permissive license. We hope our work offers a research baseline and practical insights for Korean LLM research. The model can be found at: https://huggingface.co/kifai/GECKO-7B

Related papers

• Dictionary Insertion Prompting for Multilingual Reasoning on Multilingual Large Language Models [52.00446751692225]
  We present a novel and simple yet effective method called textbfDictionary textbfInsertion textbfPrompting (textbfDIP) When providing a non-English prompt, DIP looks up a word dictionary and inserts words' English counterparts into the prompt for LLMs. It then enables better translation into English and better English model thinking steps which leads to obviously better results.
  arXiv  Detail & Related papers  (2024-11-02T05:10:50Z)
• Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMs [7.924819546105335]
  We propose Open Ko-LLM Leaderboard2, an improved version of the earlier Open Ko-LLM Leaderboard. The original benchmarks are entirely replaced with new tasks that are more closely aligned with real-world capabilities. Four new native Korean benchmarks are introduced to better reflect the distinct characteristics of the Korean language.
  arXiv  Detail & Related papers  (2024-10-16T10:49:22Z)
• HyperCLOVA X Technical Report [119.94633129762133]
  We introduce HyperCLOVA X, a family of large language models (LLMs) tailored to the Korean language and culture. HyperCLOVA X was trained on a balanced mix of Korean, English, and code data, followed by instruction-tuning with high-quality human-annotated datasets. The model is evaluated across various benchmarks, including comprehensive reasoning, knowledge, commonsense, factuality, coding, math, chatting, instruction-following, and harmlessness, in both Korean and English.
  arXiv  Detail & Related papers  (2024-04-02T13:48:49Z)
• Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models [9.359647125218359]
  This report introduces textttEEVE-Korean-v1.0, a Korean adaptation of large language models. Our method can significantly boost non-English proficiency within just 2 billion tokens.
  arXiv  Detail & Related papers  (2024-02-22T17:12:39Z)
• KMMLU: Measuring Massive Multitask Language Understanding in Korean [32.06346608507584]
  We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean benchmarks are translated from existing English benchmarks, KMMLU is collected from original Korean exams.
  arXiv  Detail & Related papers  (2024-02-18T11:41:07Z)
• Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages [51.301942056881146]
  We investigate how large language models (LLMs) function as rerankers in cross-lingual information retrieval systems for African languages. Our implementation covers English and four African languages (Hausa, Somali, Swahili, and Yoruba) We examine cross-lingual reranking with queries in English and passages in the African languages.
  arXiv  Detail & Related papers  (2023-12-26T18:38:54Z)
• Baichuan 2: Open Large-scale Language Models [51.56361715162972]
  We present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval.
  arXiv  Detail & Related papers  (2023-09-19T04:13:22Z)
• Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback [61.83548032416181]
  We present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research.
  arXiv  Detail & Related papers  (2023-07-29T18:01:46Z)
• Chain-of-Dictionary Prompting Elicits Translation in Large Language Models [100.47154959254937]
  Large language models (LLMs) have shown surprisingly good performance in multilingual neural machine translation (MNMT) We present a novel method, CoD, which augments LLMs with prior knowledge with the chains of multilingual dictionaries for a subset of input words to elicit translation abilities.
  arXiv  Detail & Related papers  (2023-05-11T05:19:47Z)
• KLUE: Korean Language Understanding Evaluation [43.94952771238633]
  We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks. We build all of the tasks from scratch from diverse source corpora while respecting copyrights.
  arXiv  Detail & Related papers  (2021-05-20T11:40:30Z)
• KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding [6.414554168135807]
  KoreALBERT is a monolingual ALBERT model specifically for Korean language understanding. Our pretrained KoreALBERT outperforms its BERT counterpart on 6 different NLU tasks.
  arXiv  Detail & Related papers  (2021-01-27T12:48:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.