German FinBERT: A German Pre-trained Language Model
- URL: http://arxiv.org/abs/2311.08793v1
- Date: Wed, 15 Nov 2023 09:07:29 GMT
- Title: German FinBERT: A German Pre-trained Language Model
- Authors: Moritz Scherrmann
- Abstract summary: This study presents German FinBERT, a novel pre-trained German language model tailored for financial textual data.
The model is trained through a comprehensive pre-training process, leveraging a substantial corpus comprising financial reports, ad-hoc announcements and news related to German companies.
I evaluate the performance of German FinBERT on downstream tasks, specifically sentiment prediction, topic recognition and question answering against generic German language models.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This study presents German FinBERT, a novel pre-trained German language model
tailored for financial textual data. The model is trained through a
comprehensive pre-training process, leveraging a substantial corpus comprising
financial reports, ad-hoc announcements and news related to German companies.
The corpus size is comparable to the data sets commonly used for training
standard BERT models. I evaluate the performance of German FinBERT on
downstream tasks, specifically sentiment prediction, topic recognition and
question answering against generic German language models. My results
demonstrate improved performance on finance-specific data, indicating the
efficacy of German FinBERT in capturing domain-specific nuances. The presented
findings suggest that German FinBERT holds promise as a valuable tool for
financial text analysis, potentially benefiting various applications in the
financial domain.
Related papers
- No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks [73.11935193630823]
ICE-PIXIU uniquely integrates a spectrum of Chinese tasks, alongside translated and original English datasets.
It provides unrestricted access to diverse model variants, a compilation of diverse cross-lingual and multi-modal instruction data, and an evaluation benchmark with expert annotations.
arXiv Detail & Related papers (2024-03-10T16:22:20Z) - Domain-Specific Language Model Post-Training for Indonesian Financial
NLP [1.8377013498056056]
BERT and IndoBERT have achieved impressive performance in several NLP tasks.
We focus on financial domain and Indonesian language, where we perform post-training on pre-trained IndoBERT for financial domain.
arXiv Detail & Related papers (2023-10-15T05:07:08Z) - BBT-Fin: Comprehensive Construction of Chinese Financial Domain
Pre-trained Language Model, Corpus and Benchmark [12.457193087920183]
We introduce BBT-FinT5, a new Chinese financial pre-training language model based on the T5 model.
To support this effort, we have built BBT-FinCorpus, a large-scale financial corpus with approximately 300GB of raw text from four different sources.
arXiv Detail & Related papers (2023-02-18T22:20:37Z) - WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model
for Financial Domain [42.093876880881886]
We propose a novel domain specific Financial LANGuage model (FLANG)
It uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective.
Our models, code and benchmark data are publicly available on Github and Huggingface.
arXiv Detail & Related papers (2022-10-31T18:35:18Z) - From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early
Modern French [57.886210204774834]
We present our efforts to develop NLP tools for Early Modern French (historical French from the 16$textth$ to the 18$textth$ centuries).
We present the $textFreEM_textmax$ corpus of Early Modern French and D'AlemBERT, a RoBERTa-based language model trained on $textFreEM_textmax$.
arXiv Detail & Related papers (2022-02-18T22:17:22Z) - GottBERT: a pure German Language Model [0.0]
No German single language RoBERTa model is yet published, which we introduce in this work (GottBERT)
In an evaluation we compare its performance on the two Named Entity Recognition (NER) tasks Conll 2003 and GermEval 2014 as well as on the text classification tasks GermEval 2018 (fine and coarse) and GNAD with existing German single language BERT models and two multilingual ones.
GottBERT was successfully pre-trained on a 256 core TPU pod using the RoBERTa BASE architecture.
arXiv Detail & Related papers (2020-12-03T17:45:03Z) - InfoBERT: Improving Robustness of Language Models from An Information
Theoretic Perspective [84.78604733927887]
Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks.
Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks.
We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
arXiv Detail & Related papers (2020-10-05T20:49:26Z) - Evaluating German Transformer Language Models with Syntactic Agreement
Tests [63.760423764010376]
Pre-trained transformer language models (TLMs) have recently refashioned natural language processing (NLP)
We design numerous agreement tasks, some of which consider peculiarities of the German language.
Our experimental results show that state-of-the-art German TLMs generally perform well on agreement tasks.
arXiv Detail & Related papers (2020-07-07T20:01:42Z) - FinBERT: A Pretrained Language Model for Financial Communications [25.900063840368347]
There is no pretrained finance specific language models available.
We address the need by pretraining a financial domain specific BERT models, FinBERT, using a large scale of financial communication corpora.
Experiments on three financial sentiment classification tasks confirm the advantage of FinBERT over generic domain BERT model.
arXiv Detail & Related papers (2020-06-15T02:51:06Z) - Revisiting Pre-Trained Models for Chinese Natural Language Processing [73.65780892128389]
We revisit Chinese pre-trained language models to examine their effectiveness in a non-English language.
We also propose a model called MacBERT, which improves upon RoBERTa in several ways.
arXiv Detail & Related papers (2020-04-29T02:08:30Z) - Coreferential Reasoning Learning for Language Representation [88.14248323659267]
We present CorefBERT, a novel language representation model that can capture the coreferential relations in context.
The experimental results show that, compared with existing baseline models, CorefBERT can achieve significant improvements consistently on various downstream NLP tasks.
arXiv Detail & Related papers (2020-04-15T03:57:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.