Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew

• URL: http://arxiv.org/abs/2309.14568v1
• Date: Mon, 25 Sep 2023 22:42:09 GMT
• Title: Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew
• Authors: Shaltiel Shmidman, Avi Shmidman, Amir David Nissan Cohen, Moshe Koppel
• Abstract summary: We present DictaLM, a large-scale language model tailored for Modern Hebrew. As a commitment to promoting research and development in the Hebrew language, we release both the foundation model and the instruct-tuned model under a Creative Commons license.
• Score: 2.1547347528250875
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present DictaLM, a large-scale language model tailored for Modern Hebrew. Boasting 7B parameters, this model is predominantly trained on Hebrew-centric data. As a commitment to promoting research and development in the Hebrew language, we release both the foundation model and the instruct-tuned model under a Creative Commons license. Concurrently, we introduce DictaLM-Rab, another foundation model geared towards Rabbinic/Historical Hebrew. These foundation models serve as ideal starting points for fine-tuning various Hebrew-specific tasks, such as instruction, Q&A, sentiment analysis, and more. This release represents a preliminary step, offering an initial Hebrew LLM model for the Hebrew NLP community to experiment with.

Related papers

• AceGPT, Localizing Large Language Models in Arabic [73.39989503874634]
  The paper proposes a comprehensive solution that includes pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities.
  arXiv  Detail & Related papers  (2023-09-21T13:20:13Z)
• DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew [2.421705925711388]
  We present DictaBERT, a new state-of-the-art pre-trained BERT model for modern Hebrew. We release three fine-tuned versions of the model, designed to perform three foundational tasks in the analysis of Hebrew texts.
  arXiv  Detail & Related papers  (2023-08-31T12:43:18Z)
• Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models [57.76998376458017]
  We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric foundation and instruction-tuned open generative large language models (LLMs) The models are based on the GPT-3 decoder-only architecture and are pretrained on a mixture of Arabic and English texts. We provide a detailed description of the training, the tuning, the safety alignment, and the evaluation of the models.
  arXiv  Detail & Related papers  (2023-08-30T17:07:17Z)
• Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All [8.964815786230686]
  We present a new pre-trained language model (PLM) for modern Hebrew, termed AlephBERTGimmel, which employs a much larger vocabulary (128K items) than standard Hebrew PLMs before. We perform a contrastive analysis of this model against all previous Hebrew PLMs (mBERT, heBERT, AlephBERT) and assess the effects of larger vocabularies on task performance. Our experiments show that larger vocabularies lead to fewer splits, and that reducing splits is better for model performance, across different tasks.
  arXiv  Detail & Related papers  (2022-11-28T10:17:35Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language [3.0663766446277845]
  We present a new pre-trained language model (PLM) for Rabbinic Hebrew, termed Berel. Berel is trained on modern Hebrew texts, which diverges substantially from Rabbinic Hebrew in terms of its lexicographical, morphological, syntactic and orthographic norms. We demonstrate the superiority of Berel on Rabbinic texts via a challenge set of Hebrew homographs.
  arXiv  Detail & Related papers  (2022-08-03T06:59:04Z)
• ParaShoot: A Hebrew Question Answering Dataset [22.55706811131828]
  ParaShoot is the first question-answering dataset in modern Hebrew. We provide the first baseline results using recently-released BERT-style models for Hebrew.
  arXiv  Detail & Related papers  (2021-09-23T11:59:38Z)
• AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With [7.345047237652976]
  Large Pre-trained Language Models (PLMs) have become ubiquitous in the development of language understanding technology. While advances reported for English using PLMs are unprecedented, reported advances using PLMs in Hebrew are few and far between.
  arXiv  Detail & Related papers  (2021-04-08T20:51:29Z)
• Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT [129.99918589405675]
  We present an effective approach that reuses an LM that is pretrained only on the high-resource language. The monolingual LM is fine-tuned on both languages and is then used to initialize a UNMT model. Our approach, RE-LM, outperforms a competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English-Albanian (En-Sq)
  arXiv  Detail & Related papers  (2020-09-16T11:37:10Z)
• Revisiting Pre-Trained Models for Chinese Natural Language Processing [73.65780892128389]
  We revisit Chinese pre-trained language models to examine their effectiveness in a non-English language. We also propose a model called MacBERT, which improves upon RoBERTa in several ways.
  arXiv  Detail & Related papers  (2020-04-29T02:08:30Z)
• A Discriminative Latent-Variable Model for Bilingual Lexicon Induction [100.76471407472599]
  We introduce a novel discriminative latent variable model for bilingual lexicon induction. Our model combines the bipartite matching dictionary prior of Haghighi et al. (2008) with a representation-based approach.
  arXiv  Detail & Related papers  (2018-08-28T14:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.