ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining

• URL: http://arxiv.org/abs/2507.06795v2
• Date: Thu, 10 Jul 2025 07:05:41 GMT
• Title: ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining
• Authors: Seonwu Kim, Yohan Na, Kihun Kim, Hanhee Cho, Geun Lim, Mintae Kim, Seongik Park, Ki Hyun Kim, Youngsub Han, Byoung-Ki Jeon,
• Abstract summary: Open-source large language models (LLMs) have expanded opportunities for enterprise applications.<n>Many organizations still lack the infrastructure to deploy and maintain large-scale models.<n>Small LLMs (sLLMs) have become a practical alternative, despite their inherent performance limitations.
• Score: 3.23679178774858
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The emergence of open-source large language models (LLMs) has expanded opportunities for enterprise applications; however, many organizations still lack the infrastructure to deploy and maintain large-scale models. As a result, small LLMs (sLLMs) have become a practical alternative, despite their inherent performance limitations. While Domain Adaptive Continual Pretraining (DACP) has been previously explored as a method for domain adaptation, its utility in commercial applications remains under-examined. In this study, we validate the effectiveness of applying a DACP-based recipe across diverse foundation models and service domains. Through extensive experiments and real-world evaluations, we demonstrate that DACP-applied sLLMs achieve substantial gains in target domain performance while preserving general capabilities, offering a cost-efficient and scalable solution for enterprise-level deployment.

Related papers

• Adversarial Data Augmentation for Single Domain Generalization via Lyapunov Exponent-Guided Optimization [6.619253289031494]
  Single Domain Generalization aims to develop models capable of generalizing to unseen target domains using only one source domain.<n>We propose LEAwareSGD, a novel Lyapunov Exponent (LE)-guided optimization approach inspired by dynamical systems theory.<n>Experiments on PACS, OfficeHome, and DomainNet demonstrate that LEAwareSGD yields substantial generalization gains.
  arXiv  Detail & Related papers  (2025-07-06T09:03:08Z)
• MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASR [59.83547898874152]
  We introduce a sample-efficient, two-stage adaptation approach that integrates self-supervised learning with semi-supervised techniques.<n>MSDA is designed to enhance the robustness and generalization of ASR models.<n>We demonstrate that Meta PL can be applied effectively to ASR tasks, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2025-05-30T14:46:05Z)
• Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language [4.5224851085910585]
  Domain continual pretraining (DAPT) is a state-of-the-art technique that further trains a language model (LM) on its pretraining task.<n>This paper introduces ICL-augmented pretraining (ICL-APT) that leverages in-adaptive learning (ICL) and kNN to augment target data with domain-related and in-domain texts.<n>Our results show that the best configuration of ICL-APT performed better than the state-of-the-art DAPT by 28.7% and requires almost 4 times less GPU-computing time.
  arXiv  Detail & Related papers  (2025-04-28T14:49:00Z)
• CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey [38.281260447611395]
  This survey systematically explores the applications of Contrastive Language-Image Pretraining (CLIP) in domain generalization (DG) and domain adaptation (DA)<n>CLIP offers powerful zero-shot capabilities that allow models to perform effectively in unseen domains.<n>Key challenges, including overfitting, domain diversity, and computational efficiency, are addressed.
  arXiv  Detail & Related papers  (2025-04-19T12:27:24Z)
• A Survey of Direct Preference Optimization [103.59317151002693]
  Large Language Models (LLMs) have demonstrated unprecedented generative capabilities.<n>Their alignment with human values remains critical for ensuring helpful and harmless deployments.<n>Direct Preference Optimization (DPO) has recently gained prominence as a streamlined alternative.
  arXiv  Detail & Related papers  (2025-03-12T08:45:15Z)
• Demystifying Domain-adaptive Post-training for Financial LLMs [79.581577578952]
  FINDAP is a systematic and fine-grained investigation into domain adaptive post-training of large language models (LLMs)<n>Our approach consists of four key components: FinCap, FinRec, FinTrain and FinEval.<n>The resulting model, Llama-Fin, achieves state-of-the-art performance across a wide range of financial tasks.
  arXiv  Detail & Related papers  (2025-01-09T04:26:15Z)
• On Domain-Adaptive Post-Training for Multimodal Large Language Models [72.67107077850939]
  This paper systematically investigates domain adaptation of MLLMs via post-training.<n>We focus on data synthesis, training pipeline, and task evaluation.<n>We conduct experiments in high-impact domains such as biomedicine, food, and remote sensing.
  arXiv  Detail & Related papers  (2024-11-29T18:42:28Z)
• Exploring Language Model Generalization in Low-Resource Extractive QA [57.14068405860034]
  We investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift.<n>We devise a series of experiments to explain the performance gap empirically.
  arXiv  Detail & Related papers  (2024-09-27T05:06:43Z)
• DOLLmC: DevOps for Large Language model Customization [0.0]
  This research aims to establish a scalable and efficient framework for LLM customization. We propose a robust framework that enhances continuous learning, seamless deployment, and rigorous version control of LLMs.
  arXiv  Detail & Related papers  (2024-05-19T15:20:27Z)
• Efficient Continual Pre-training for Building Domain Specific Large Language Models [8.799785664150255]
  Large language models (LLMs) have demonstrated remarkable open-domain capabilities. Traditionally, LLMs tailored for a domain are trained from scratch to excel at handling domain-specific tasks. We introduce FinPythia-6.9B, developed through domain-adaptive continual pre-training on the financial domain.
  arXiv  Detail & Related papers  (2023-11-14T21:19:14Z)
• Open-Set Domain Adaptation with Visual-Language Foundation Models [51.49854335102149]
  Unsupervised domain adaptation (UDA) has proven to be very effective in transferring knowledge from a source domain to a target domain with unlabeled data. Open-set domain adaptation (ODA) has emerged as a potential solution to identify these classes during the training phase.
  arXiv  Detail & Related papers  (2023-07-30T11:38:46Z)
• Universal Source-Free Domain Adaptation [57.37520645827318]
  We propose a novel two-stage learning process for domain adaptation. In the Procurement stage, we aim to equip the model for future source-free deployment, assuming no prior knowledge of the upcoming category-gap and domain-shift. In the Deployment stage, the goal is to design a unified adaptation algorithm capable of operating across a wide range of category-gaps.
  arXiv  Detail & Related papers  (2020-04-09T07:26:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.