TourBERT: A pretrained language model for the tourism industry
- URL: http://arxiv.org/abs/2201.07449v1
- Date: Wed, 19 Jan 2022 07:24:30 GMT
- Title: TourBERT: A pretrained language model for the tourism industry
- Authors: Veronika Arefieva and Roman Egger
- Abstract summary: Bidirectional Representations from Transformers (BERT) is currently one of the most important and state-of-the-art models for natural language.
We present TourBERT, a pretrained language model for tourism.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The Bidirectional Encoder Representations from Transformers (BERT) is
currently one of the most important and state-of-the-art models for natural
language. However, it has also been shown that for domain-specific tasks it is
helpful to pretrain BERT on a domain-specific corpus. In this paper, we present
TourBERT, a pretrained language model for tourism. We describe how TourBERT was
developed and evaluated. The evaluations show that TourBERT is outperforming
BERT in all tourism-specific tasks.
Related papers
- TourLLM: Enhancing LLMs with Tourism Knowledge [3.034710104407876]
We construct a supervised fine-tuning dataset for the culture and tourism domain, named Cultour.
This dataset consists of three parts: tourism knowledge base QA data, travelogues data, and tourism diversity QA data.
We propose TourLLM, a Qwen-based model supervised fine-tuned with Cultour, to improve the quality of the information about attractions and travel planning.
arXiv Detail & Related papers (2024-06-18T09:15:46Z) - Harnessing the Power of BERT in the Turkish Clinical Domain: Pretraining
Approaches for Limited Data Scenarios [0.0]
General Turkish BERT model (BERTurk) and TurkRadBERT-task v1, both of which utilize knowledge from a substantial general-domain corpus, demonstrate the best overall performance.
Our results underscore the significance of domain-specific vocabulary during pre-training for enhancing model performance.
arXiv Detail & Related papers (2023-05-05T18:39:07Z) - BERTino: an Italian DistilBERT model [0.0]
We present BERTino, a DistilBERT model which proposes to be the first lightweight alternative to the BERT architecture specific for the Italian language.
We evaluate BERTino on the Italian ISDT, Italian ParTUT, Italian WikiNER and multiclass classification tasks, obtaining F1 scores comparable to those obtained by a BERTBASE with a remarkable improvement in training and inference speed.
arXiv Detail & Related papers (2023-03-31T15:07:40Z) - NarrowBERT: Accelerating Masked Language Model Pretraining and Inference [50.59811343945605]
We propose NarrowBERT, a modified transformer encoder that increases the throughput for masked language model pretraining by more than $2times$.
NarrowBERT sparsifies the transformer model such that the self-attention queries and feedforward layers only operate on the masked tokens of each sentence during pretraining.
We show that NarrowBERT increases the throughput at inference time by as much as $3.5times$ with minimal (or no) performance degradation on sentence encoding tasks like MNLI.
arXiv Detail & Related papers (2023-01-11T23:45:50Z) - Embodied BERT: A Transformer Model for Embodied, Language-guided Visual
Task Completion [69.04196388421649]
We present Embodied BERT (EmBERT), a transformer-based model which can attend to high-dimensional, multi-modal inputs across long temporal horizons for language-conditioned task completion.
We achieve competitive performance on the ALFRED benchmark, and EmBERT marks the first transformer-based model to successfully handle the long-horizon, dense, multi-modal histories of ALFRED.
arXiv Detail & Related papers (2021-08-10T21:24:05Z) - VisBERT: Hidden-State Visualizations for Transformers [66.86452388524886]
We present VisBERT, a tool for visualizing the contextual token representations within BERT for the task of (multi-hop) Question Answering.
VisBERT enables users to get insights about the model's internal state and to explore its inference steps or potential shortcomings.
arXiv Detail & Related papers (2020-11-09T15:37:43Z) - It's not Greek to mBERT: Inducing Word-Level Translations from
Multilingual BERT [54.84185432755821]
multilingual BERT (mBERT) learns rich cross-lingual representations, that allow for transfer across languages.
We study the word-level translation information embedded in mBERT and present two simple methods that expose remarkable translation capabilities with no fine-tuning.
arXiv Detail & Related papers (2020-10-16T09:49:32Z) - InfoBERT: Improving Robustness of Language Models from An Information
Theoretic Perspective [84.78604733927887]
Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks.
Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks.
We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
arXiv Detail & Related papers (2020-10-05T20:49:26Z) - Domain-Specific Language Model Pretraining for Biomedical Natural
Language Processing [73.37262264915739]
We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains.
Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
arXiv Detail & Related papers (2020-07-31T00:04:15Z) - lamBERT: Language and Action Learning Using Multimodal BERT [0.1942428068361014]
This study proposes the language and action learning using multimodal BERT (lamBERT) model.
Experiment is conducted in a grid environment that requires language understanding for the agent to act properly.
The lamBERT model obtained higher rewards in multitask settings and transfer settings when compared to other models.
arXiv Detail & Related papers (2020-04-15T13:54:55Z) - What the [MASK]? Making Sense of Language-Specific BERT Models [39.54532211263058]
This paper presents the current state of the art in language-specific BERT models.
Our aim is to provide an overview of the commonalities and differences between Language-language-specific BERT models and mBERT models.
arXiv Detail & Related papers (2020-03-05T20:42:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.