Thai Financial Domain Adaptation of THaLLE -- Technical Report

• URL: http://arxiv.org/abs/2411.18242v1
• Date: Wed, 27 Nov 2024 11:30:00 GMT
• Title: Thai Financial Domain Adaptation of THaLLE -- Technical Report
• Authors: KBTG Labs, Atthakorn Petchsod, Pornchanan Balee, Danupat Khamnuansin, Anuruth Lertpiya, Chanatip Saetia, Tawunrat Chalothorn, Thadpong Pongthawornkamol, Monchai Lertsutthiwong,
• Abstract summary: Large Language Models (LLMs) excel in general tasks but struggle with domain-specific challenges. We developed a Thai Financial LLM using the Investment Consultant (IC) exam dataset from the Stock Exchange of Thailand. The model achieved scores of 72%, 72%, and 84% on IC exam levels P1, P2, and P3, respectively.
• Score: 0.0
• License:
• Abstract: Large Language Models (LLMs) excel in general tasks but struggle with domain-specific challenges, such as specialized terminology and localized regulations. Existing financial LLMs, like FinGPT and BloombergGPT, lack support for the Thai financial domain. We developed a Thai Financial LLM using the Investment Consultant (IC) exam dataset from the Stock Exchange of Thailand. To address dataset limitations, we applied data augmentation, ReLoRA for efficient training, Continued Pretraining (CPT) for domain knowledge, and Rank-Stabilized LoRA (rsLoRA) for fine-tuning. Supervised Fine-Tuning (SFT) simulated exam scenarios, while Direct Preference Optimization (DPO) refined the model using feedback. The model achieved scores of 72%, 72%, and 84% on IC exam levels P1, P2, and P3, respectively, demonstrating its effectiveness in Thai financial advisory tasks and its potential for specialized applications.

Related papers

• Demystifying Domain-adaptive Post-training for Financial LLMs [79.581577578952]
  FINDAP is a systematic and fine-grained investigation into domain adaptive post-training of large language models (LLMs) Our approach consists of four key components: FinCap, FinRec, FinTrain and FinEval. The resulting model, Llama-Fin, achieves state-of-the-art performance across a wide range of financial tasks.
  arXiv  Detail & Related papers  (2025-01-09T04:26:15Z)
• Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models [22.594428755214356]
  "Golden Touchstone" is the first comprehensive bilingual benchmark for financial LLMs. benchmarks include a variety of financial tasks aimed at thoroughly assessing models' language understanding and generation capabilities. We open-sourced Touchstone-GPT, a financial LLM trained through continual pre-training and financial instruction tuning.
  arXiv  Detail & Related papers  (2024-11-09T20:09:11Z)
• BreakGPT: Leveraging Large Language Models for Predicting Asset Price Surges [55.2480439325792]
  This paper introduces BreakGPT, a novel large language model (LLM) architecture adapted specifically for time series forecasting and the prediction of sharp upward movements in asset prices. We showcase BreakGPT as a promising solution for financial forecasting with minimal training and as a strong competitor for capturing both local and global temporal dependencies.
  arXiv  Detail & Related papers  (2024-11-09T05:40:32Z)
• CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models [61.324062412648075]
  CFinBench is an evaluation benchmark for assessing the financial knowledge of large language models (LLMs) under Chinese context. It comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment. The results show that GPT4 and some Chinese-oriented models lead the benchmark, with the highest average accuracy being 60.16%.
  arXiv  Detail & Related papers  (2024-07-02T14:34:36Z)
• D\'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English [67.48541936784501]
  Tois'on de Oro is the first framework that establishes instruction datasets, finetuned LLMs, and evaluation benchmark for financial LLMs in Spanish joint with English. We construct a rigorously curated bilingual instruction dataset including over 144K Spanish and English samples from 15 datasets covering 7 tasks. We evaluate our model and existing LLMs using FLARE-ES, the first comprehensive bilingual evaluation benchmark with 21 datasets covering 9 tasks.
  arXiv  Detail & Related papers  (2024-02-12T04:50:31Z)
• Revolutionizing Finance with LLMs: An Overview of Applications and Insights [45.660896719456886]
  Large Language Models (LLMs) like ChatGPT have seen considerable advancements and have been applied in diverse fields. These models are being utilized for automating financial report generation, forecasting market trends, analyzing investor sentiment, and offering personalized financial advice.
  arXiv  Detail & Related papers  (2024-01-22T01:06:17Z)
• Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models [2.5749046466046903]
  We show that financial language models (PLMs) are not pretrained on sufficiently diverse financial data. To address this issue, we trained the Financial Language Model (FiLM) on these diverse datasets. Our experimental results confirm that FiLM outperforms not only existing financial PLMs but also general domain PLMs.
  arXiv  Detail & Related papers  (2023-10-20T07:04:08Z)
• Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams [26.318005637849915]
  This study aims at assessing the financial reasoning capabilities of Large Language Models (LLMs) We leverage mock exam questions of the Chartered Financial Analyst (CFA) Program to conduct a comprehensive evaluation of ChatGPT and GPT-4. We present an in-depth analysis of the models' performance and limitations, and estimate whether they would have a chance at passing the CFA exams.
  arXiv  Detail & Related papers  (2023-10-12T19:28:57Z)
• InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning [19.22852919096857]
  We present a new financial domain large language model, InvestLM, tuned on LLaMA-65B (Touvron et al., 2023) Inspired by less-is-more-for-alignment, we manually curate a small yet diverse instruction dataset, covering a wide range of financial related topics. InvestLM shows strong capabilities in understanding financial text and provides helpful responses to investment related questions.
  arXiv  Detail & Related papers  (2023-09-15T02:59:31Z)
• PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance [63.51545277822702]
  PIXIU is a comprehensive framework including the first financial large language model (LLMs) based on fine-tuning LLaMA with instruction data. We propose FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. We conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks.
  arXiv  Detail & Related papers  (2023-06-08T14:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.