What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

• URL: http://arxiv.org/abs/2109.04650v1
• Date: Fri, 10 Sep 2021 03:32:19 GMT
• Title: What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• Authors: Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung
• Abstract summary: GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. We introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. We show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline.
• Score: 16.596023525331862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

Related papers

• Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study [11.503751895052572]
  We introduce Llama-3-Motif, a language model consisting of 102 billion parameters, specifically designed to enhance Korean capabilities while retaining strong performance in English.<n>Llama-3-Motif employs advanced training techniques, including LlamaPro and Masked Structure Growth, to effectively scale the model without altering its core Transformer architecture.<n>Using the MoAI platform for efficient training across hyperscale GPU clusters, we optimized Llama-3-Motif using a carefully curated dataset that maintains a balanced ratio of Korean and English data.
  arXiv  Detail & Related papers  (2025-09-04T07:56:24Z)
• HyperCLOVA X THINK Technical Report [0.0]
  We introduce HyperCLOVA X THINK, the first reasoning-focused large language model in the HyperCLOVA X family.<n>It pre-trained on roughly $6$ trillion high-quality Korean, and English tokens, augmented with targeted synthetic Korean data.<n>It delivers competitive performance against similarly sized models on Korea-focused benchmarks.
  arXiv  Detail & Related papers  (2025-06-27T17:23:12Z)
• Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs [123.25404278506585]
  We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs) To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters.
  arXiv  Detail & Related papers  (2025-04-10T15:41:51Z)
• HyperCLOVA X Technical Report [119.94633129762133]
  We introduce HyperCLOVA X, a family of large language models (LLMs) tailored to the Korean language and culture. HyperCLOVA X was trained on a balanced mix of Korean, English, and code data, followed by instruction-tuning with high-quality human-annotated datasets. The model is evaluated across various benchmarks, including comprehensive reasoning, knowledge, commonsense, factuality, coding, math, chatting, instruction-following, and harmlessness, in both Korean and English.
  arXiv  Detail & Related papers  (2024-04-02T13:48:49Z)
• Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models [9.359647125218359]
  This report introduces textttEEVE-Korean-v1.0, a Korean adaptation of large language models. Our method can significantly boost non-English proficiency within just 2 billion tokens.
  arXiv  Detail & Related papers  (2024-02-22T17:12:39Z)
• In-Context Language Learning: Architectures and Algorithms [73.93205821154605]
  We study ICL through the lens of a new family of model problems we term in context language learning (ICLL) We evaluate a diverse set of neural sequence models on regular ICLL tasks.
  arXiv  Detail & Related papers  (2024-01-23T18:59:21Z)
• PanGu-{\Sigma}: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing [64.53242758625922]
  PanGu-Sigma is trained on a cluster of Ascend 910 AI processors and MindSpore framework. It provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks.
  arXiv  Detail & Related papers  (2023-03-20T03:39:27Z)
• mGPT: Few-Shot Learners Go Multilingual [1.4354798873010843]
  This paper introduces two autoregressive GPT-like models with 1.3 billion and 13 billion parameters trained on 60 languages. We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism. The resulting models show performance on par with the recently released XGLM models by Facebook.
  arXiv  Detail & Related papers  (2022-04-15T13:02:33Z)
• PaLM: Scaling Language Modeling with Pathways [180.69584031908113]
  We trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2022-04-05T16:11:45Z)
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model [35.75234515196426]
  We present details on the training of the largest monolithic transformer based language model, Megatron-Turing NLG 530B (MT-NLG) with 530 billion parameters. We show that MT-NLG achieves superior zero-, one-, and few-shot learning accuracies on several NLP benchmarks and establishes new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-01-28T08:59:57Z)
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation [58.31465205357637]
  We present our practice on training large-scale autoregressive language models named PanGu-$alpha$, with up to 200 billion parameters. PanGu-$alpha$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors.
  arXiv  Detail & Related papers  (2021-04-26T06:59:36Z)
• CPM: A Large-scale Generative Chinese Pre-trained Language Model [76.65305358932393]
  We release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning.
  arXiv  Detail & Related papers  (2020-12-01T11:32:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.