SCALE: Scaling up the Complexity for Advanced Language Model Evaluation
- URL: http://arxiv.org/abs/2306.09237v2
- Date: Fri, 1 Sep 2023 18:00:57 GMT
- Title: SCALE: Scaling up the Complexity for Advanced Language Model Evaluation
- Authors: Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias St\"urmer,
Ilias Chalkidis, Daniel E. Ho, Joel Niklaus
- Abstract summary: We introduce a novel NLP benchmark that poses challenges to current Large Language Models (LLMs)
Our benchmark comprises diverse legal NLP datasets from the Swiss legal system.
As part of our study, we evaluate several pre-trained multilingual language models on our benchmark to establish strong baselines as a point of reference.
- Score: 19.339580164451256
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent strides in Large Language Models (LLMs) have saturated many NLP
benchmarks (even professional domain-specific ones), emphasizing the need for
novel, more challenging novel ones to properly assess LLM capabilities. In this
paper, we introduce a novel NLP benchmark that poses challenges to current LLMs
across four key dimensions: processing long documents (up to 50K tokens),
utilizing domain specific knowledge (embodied in legal texts), multilingual
understanding (covering five languages), and multitasking (comprising legal
document to document Information Retrieval, Court View Generation, Leading
Decision Summarization, Citation Extraction, and eight challenging Text
Classification tasks). Our benchmark comprises diverse legal NLP datasets from
the Swiss legal system, allowing for a comprehensive study of the underlying
Non-English, inherently multilingual, federal legal system. Despite recent
advances, efficiently processing long documents for intense review/analysis
tasks remains an open challenge for language models. Also, comprehensive,
domain-specific benchmarks requiring high expertise to develop are rare, as are
multilingual benchmarks. This scarcity underscores our contribution's value,
considering most public models are trained predominantly on English corpora,
while other languages remain understudied, particularly for practical
domain-specific NLP tasks. Our benchmark allows for testing and advancing the
state-of-the-art LLMs. As part of our study, we evaluate several pre-trained
multilingual language models on our benchmark to establish strong baselines as
a point of reference. Despite the large size of our datasets (tens to hundreds
of thousands of examples), existing publicly available models struggle with
most tasks, even after in-domain pretraining. We publish all resources
(benchmark suite, pre-trained models, code) under a fully permissive open CC
BY-SA license.
Related papers
- Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers [81.47046536073682]
We present a review and provide a unified perspective to summarize the recent progress as well as emerging trends in multilingual large language models (MLLMs) literature.
We hope our work can provide the community with quick access and spur breakthrough research in MLLMs.
arXiv Detail & Related papers (2024-04-07T11:52:44Z) - BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models [141.21603469555225]
Large language models (LLMs) have achieved dramatic proficiency over NLP tasks with normal length.
We propose BAMBOO, a multi-task long context benchmark.
It consists of 10 datasets from 5 different long text understanding tasks.
arXiv Detail & Related papers (2023-09-23T11:36:15Z) - Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars.
We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English.
Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
arXiv Detail & Related papers (2023-06-20T08:27:47Z) - Multilingual Large Language Models Are Not (Yet) Code-Switchers [41.47534626749588]
Large Language Models (LLMs) have recently shown great capabilities in a wide range of tasks.
The practice of alternating languages within an utterance remains relatively uncharted.
We argue that current "multilingualism" in LLMs does not inherently imply proficiency with code-switching texts.
arXiv Detail & Related papers (2023-05-23T16:50:48Z) - Beyond Static Models and Test Sets: Benchmarking the Potential of
Pre-trained Models Across Tasks and Languages [15.373725507698591]
We argue that this makes the existing practices in multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape.
We propose that the recent work done in Performance Prediction for NLP tasks can serve as a potential solution in fixing benchmarking in Multilingual NLP.
We compare performance prediction with translating test data with a case study on four different multilingual datasets, and observe that these methods can provide reliable estimates of the performance that are often on-par with the translation based approaches.
arXiv Detail & Related papers (2022-05-12T20:42:48Z) - IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and
Languages [87.5457337866383]
We introduce the Image-Grounded Language Understanding Evaluation benchmark.
IGLUE brings together visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages.
We find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks.
arXiv Detail & Related papers (2022-01-27T18:53:22Z) - LAWDR: Language-Agnostic Weighted Document Representations from
Pre-trained Models [8.745407715423992]
Cross-lingual document representations enable language understanding in multilingual contexts.
Large pre-trained language models such as BERT, XLM and XLM-RoBERTa have achieved great success when fine-tuned on sentence-level downstream tasks.
arXiv Detail & Related papers (2021-06-07T07:14:00Z) - UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks.
Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages.
We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
arXiv Detail & Related papers (2020-12-31T11:37:28Z) - KLEJ: Comprehensive Benchmark for Polish Language Understanding [4.702729080310267]
We introduce a comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard.
We also release HerBERT, a Transformer-based model trained specifically for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks.
arXiv Detail & Related papers (2020-05-01T21:55:40Z) - XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating
Cross-lingual Generalization [128.37244072182506]
Cross-lingual TRansfer Evaluation of Multilinguals XTREME is a benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks.
We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models.
arXiv Detail & Related papers (2020-03-24T19:09:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.