FETILDA: An Effective Framework For Fin-tuned Embeddings For Long
Financial Text Documents
- URL: http://arxiv.org/abs/2206.06952v1
- Date: Tue, 14 Jun 2022 16:14:14 GMT
- Title: FETILDA: An Effective Framework For Fin-tuned Embeddings For Long
Financial Text Documents
- Authors: Bolun "Namir" Xia, Vipula D. Rawte, Mohammed J. Zaki, Aparna Gupta
- Abstract summary: We propose and implement a deep learning framework that splits long documents into chunks and utilize pre-trained LMs to process and aggregate the chunks into vector representations.
We evaluate our framework on a collection of 10-K public disclosure reports from US banks, and another dataset of reports submitted by US companies.
- Score: 14.269860621624394
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Unstructured data, especially text, continues to grow rapidly in various
domains. In particular, in the financial sphere, there is a wealth of
accumulated unstructured financial data, such as the textual disclosure
documents that companies submit on a regular basis to regulatory agencies, such
as the Securities and Exchange Commission (SEC). These documents are typically
very long and tend to contain valuable soft information about a company's
performance. It is therefore of great interest to learn predictive models from
these long textual documents, especially for forecasting numerical key
performance indicators (KPIs). Whereas there has been a great progress in
pre-trained language models (LMs) that learn from tremendously large corpora of
textual data, they still struggle in terms of effective representations for
long documents. Our work fills this critical need, namely how to develop better
models to extract useful information from long textual documents and learn
effective features that can leverage the soft financial and risk information
for text regression (prediction) tasks. In this paper, we propose and implement
a deep learning framework that splits long documents into chunks and utilizes
pre-trained LMs to process and aggregate the chunks into vector
representations, followed by self-attention to extract valuable document-level
features. We evaluate our model on a collection of 10-K public disclosure
reports from US banks, and another dataset of reports submitted by US
companies. Overall, our framework outperforms strong baseline methods for
textual modeling as well as a baseline regression model using only numerical
data. Our work provides better insights into how utilizing pre-trained
domain-specific and fine-tuned long-input LMs in representing long documents
can improve the quality of representation of textual data, and therefore, help
in improving predictive analyses.
Related papers
- LongWanjuan: Towards Systematic Measurement for Long Text Quality [102.46517202896521]
LongWanjuan is a dataset specifically tailored to enhance the training of language models for long-text tasks with over 160B tokens.
In LongWanjuan, we categorize long texts into holistic, aggregated, and chaotic types, enabling a detailed analysis of long-text quality.
We devise a data mixture recipe that strategically balances different types of long texts within LongWanjuan, leading to significant improvements in model performance on long-text tasks.
arXiv Detail & Related papers (2024-02-21T07:27:18Z) - Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral.
This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
arXiv Detail & Related papers (2024-02-16T13:53:26Z) - LongFin: A Multimodal Document Understanding Model for Long Financial
Domain Documents [4.924255992661131]
We introduce LongFin, a multimodal document AI model capable of encoding up to 4K tokens.
We also propose the LongForms dataset that encapsulates several industrial challenges in financial documents.
arXiv Detail & Related papers (2024-01-26T18:23:45Z) - Large Language Model Adaptation for Financial Sentiment Analysis [2.0499240875882]
Generalist language models tend to fall short in tasks specifically tailored for finance.
Two foundation models with less than 1.5B parameters have been adapted using a wide range of strategies.
We show that small LLMs have comparable performance to larger scale models, while being more efficient in terms of parameters and data.
arXiv Detail & Related papers (2024-01-26T11:04:01Z) - Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models [29.94694305204144]
We present a novel framework for document-level in-context few-shot relation extraction.
We evaluate our framework using DocRED, the largest publicly available dataset for document-level relation extraction.
arXiv Detail & Related papers (2023-10-17T09:10:27Z) - Multimodal Document Analytics for Banking Process Automation [4.541582055558865]
The paper contributes original empirical evidence on the effectiveness and efficiency of multi-model models for document processing in the banking business.
It offers practical guidance on how to unlock this potential in day-to-day operations.
arXiv Detail & Related papers (2023-07-21T18:29:04Z) - Cross-Lingual NER for Financial Transaction Data in Low-Resource
Languages [70.25418443146435]
We propose an efficient modeling framework for cross-lingual named entity recognition in semi-structured text data.
We employ two independent datasets of SMSs in English and Arabic, each carrying semi-structured banking transaction information.
With access to only 30 labeled samples, our model can generalize the recognition of merchants, amounts, and other fields from English to Arabic.
arXiv Detail & Related papers (2023-07-16T00:45:42Z) - Exploring Large Language Model for Graph Data Understanding in Online
Job Recommendations [63.19448893196642]
We present a novel framework that harnesses the rich contextual information and semantic representations provided by large language models to analyze behavior graphs.
By leveraging this capability, our framework enables personalized and accurate job recommendations for individual users.
arXiv Detail & Related papers (2023-07-10T11:29:41Z) - Lost in the Middle: How Language Models Use Long Contexts [88.78803442320246]
We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts.
We find that performance can degrade significantly when changing the position of relevant information.
Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
arXiv Detail & Related papers (2023-07-06T17:54:11Z) - Long Document Summarization with Top-down and Bottom-up Inference [113.29319668246407]
We propose a principled inference framework to improve summarization models on two aspects.
Our framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency.
We demonstrate the effectiveness of the proposed framework on a diverse set of summarization datasets.
arXiv Detail & Related papers (2022-03-15T01:24:51Z) - NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task
Financial Forecasting [17.691653056521904]
This paper describes a numeric-oriented hierarchical transformer model to predict stock returns and financial risk using multi-modal aligned earnings calls data.
We present the results of a comprehensive evaluation of Num HTML against several state-of-the-art baselines using a real-world publicly available dataset.
arXiv Detail & Related papers (2022-01-05T10:17:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.