L3iTC at the FinLLM Challenge Task: Quantization for Financial Text Classification & Summarization

• URL: http://arxiv.org/abs/2408.03033v1
• Date: Tue, 6 Aug 2024 08:25:49 GMT
• Title: L3iTC at the FinLLM Challenge Task: Quantization for Financial Text Classification & Summarization
• Authors: Elvys Linhares Pontes, Carlos-Emiliano González-Gallardo, Mohamed Benjannet, Caryn Qu, Antoine Doucet,
• Abstract summary: FinLLM Challenge Task 2024 focused on two key areas: Task 1, financial text classification, and Task 2, financial text summarization. We fine-tuned several large language models (LLMs) to optimize performance for each task. Our models achieved third place for the financial classification task with an F1-score of 0.7543 and secured sixth place in the financial summarization task on the official test datasets.
• Score: 2.111699987679628
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This article details our participation (L3iTC) in the FinLLM Challenge Task 2024, focusing on two key areas: Task 1, financial text classification, and Task 2, financial text summarization. To address these challenges, we fine-tuned several large language models (LLMs) to optimize performance for each task. Specifically, we used 4-bit quantization and LoRA to determine which layers of the LLMs should be trained at a lower precision. This approach not only accelerated the fine-tuning process on the training data provided by the organizers but also enabled us to run the models on low GPU memory. Our fine-tuned models achieved third place for the financial classification task with an F1-score of 0.7543 and secured sixth place in the financial summarization task on the official test datasets.

Related papers

• A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification [0.8192907805418583]
  Large Language Models (LLMs) have demonstrated impressive capabilities across diverse Natural Language Processing (NLP) tasks. This study investigates the efficacy of instruction fine-tuning to enhance their performance in financial text classification tasks.
  arXiv  Detail & Related papers  (2024-11-04T18:06:36Z)
• Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications [90.67346776473241]
  Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. We introduce textitOpen-FinLLMs, a series of Financial LLMs that embed comprehensive financial knowledge into text, tables, and time-series data. We also present FinLLaVA, a multimodal LLM trained with 1.43M image-text instructions to handle complex financial data types.
  arXiv  Detail & Related papers  (2024-08-20T16:15:28Z)
• 'Finance Wizard' at the FinLLM Challenge Task: Financial Text Summarization [0.0]
  This paper documents our pipeline approach of fine-tuning a foundation model into a task-specific model for Financial Text Summarization. Our model, FinLlama3_sum, yielded commendable results, securing the third position in its category with a ROUGE-1 score of 0.521.
  arXiv  Detail & Related papers  (2024-08-07T13:31:44Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models [61.324062412648075]
  CFinBench is an evaluation benchmark for assessing the financial knowledge of large language models (LLMs) under Chinese context. It comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment. The results show that GPT4 and some Chinese-oriented models lead the benchmark, with the highest average accuracy being 60.16%.
  arXiv  Detail & Related papers  (2024-07-02T14:34:36Z)
• CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications [10.225210627594894]
  This paper presents our solution to IJCAI-2024 FinLLM challenge, investigating the capabilities of LLMs within three critical areas of financial tasks. Financial classification, financial text summarization, and single stock trading are investigated. Our approach aims to tackle these diverse tasks in a comprehensive and integrated manner, showcasing LLMs' capacity to address diverse and complex financial tasks with improved accuracy and decision-making capabilities.
  arXiv  Detail & Related papers  (2024-07-02T05:04:13Z)
• Token-Efficient Leverage Learning in Large Language Models [13.830828529873056]
  Large Language Models (LLMs) have excelled in various tasks but perform better in high-resource scenarios. Data scarcity and the inherent difficulty of adapting LLMs to specific tasks compound the challenge. We present a streamlined implement of this methodology called Token-Efficient Leverage Learning (TELL)
  arXiv  Detail & Related papers  (2024-04-01T04:39:44Z)
• FinBen: A Holistic Financial Benchmark for Large Language Models [75.09474986283394]
  FinBen is the first extensive open-source evaluation benchmark, including 36 datasets spanning 24 financial tasks. FinBen offers several key innovations: a broader range of tasks and datasets, the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) evaluation, and three novel open-source evaluation datasets for text summarization, question answering, and stock trading.
  arXiv  Detail & Related papers  (2024-02-20T02:16:16Z)
• Pre-trained Large Language Models for Financial Sentiment Analysis [10.683185786541596]
  We adapt the open-source Llama2-7B model (2023) with the supervised fine-tuning (SFT) technique. Our approach significantly outperforms the previous state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-01-10T15:27:41Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)
• PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance [63.51545277822702]
  PIXIU is a comprehensive framework including the first financial large language model (LLMs) based on fine-tuning LLaMA with instruction data. We propose FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. We conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks.
  arXiv  Detail & Related papers  (2023-06-08T14:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.