M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context
Evaluation Benchmark for Large Language Models
- URL: http://arxiv.org/abs/2310.19240v1
- Date: Mon, 30 Oct 2023 03:11:30 GMT
- Title: M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context
Evaluation Benchmark for Large Language Models
- Authors: Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li,
Lifeng Shang, Qun Liu, Kam-Fai Wong
- Abstract summary: M4LE is a benchmark for evaluating the long-sequence capability of large language models (LLMs)
M4LE is based on a diverse NLP task pool comprising 36 NLP task types and 12 domains.
We conducted a systematic evaluation on 11 well-established LLMs, especially those optimized for long-sequence inputs.
- Score: 61.06694491246026
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Managing long sequences has become an important and necessary feature for
large language models (LLMs). However, it is still an open question of how to
comprehensively and systematically evaluate the long-sequence capability of
LLMs. One of the reasons is that conventional and widely-used benchmarks mainly
consist of short sequences. In this paper, we propose M4LE, a Multi-ability,
Multi-range, Multi-task, Multi-domain benchmark for Long-context Evaluation.
M4LE is based on a diverse NLP task pool comprising 36 NLP datasets, 11 task
types and 12 domains. To alleviate the scarcity of tasks with naturally long
sequences and incorporate multiple-ability assessment, we propose an automatic
approach (but with negligible human annotations) to convert short-sequence
tasks into a unified long-sequence scenario where LLMs have to identify single
or multiple relevant spans in long contexts based on explicit or semantic
hints. Specifically, the scenario includes five different types of abilities:
(1) explicit single-span; (2) semantic single-span; (3) explicit multiple-span;
(4) semantic multiple-span; and (5) global context understanding. The resulting
samples in M4LE are evenly distributed from 1k to 8k input length. We conducted
a systematic evaluation on 11 well-established LLMs, especially those optimized
for long-sequence inputs. Our results reveal that: 1) Current LLMs struggle to
understand long context, particularly when tasks require multiple-span
attention. 2) Semantic retrieval task is more difficult for competent LLMs. 3)
Models fine-tuned on longer text with position interpolation have comparable
performance to those using Neural Tangent Kernel (NTK) aware scaling methods
without fine-tuning. We make our benchmark publicly available to encourage
future research in this challenging area.
Related papers
- NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context Window? [37.64593022203498]
NeedleBench is a framework consisting of progressively more challenging tasks for assessing bilingual long-context capabilities.
We use the framework to assess how well the leading open-source models can identify key information relevant to the question.
We propose the Ancestral Trace Challenge to mimic the complexity of logical reasoning challenges that are likely to be present in real-world long-context tasks.
arXiv Detail & Related papers (2024-07-16T17:59:06Z) - LongIns: A Challenging Long-context Instruction-based Exam for LLMs [44.51209510772957]
Long-context capabilities of large language models (LLMs) have been a hot topic in recent years.
We propose the LongIns benchmark dataset, a challenging long-context instruction-based exam for LLMs.
arXiv Detail & Related papers (2024-06-25T14:31:26Z) - Needle In A Multimodal Haystack [79.81804334634408]
We present the first benchmark specifically designed to evaluate the capability of existing MLLMs to comprehend long multimodal documents.
Our benchmark includes three types of evaluation tasks: multimodal retrieval, counting, and reasoning.
We observe that existing models still have significant room for improvement on these tasks, especially on vision-centric evaluation.
arXiv Detail & Related papers (2024-06-11T13:09:16Z) - Analyzing the Role of Semantic Representations in the Era of Large Language Models [104.18157036880287]
We investigate the role of semantic representations in the era of large language models (LLMs)
We propose an AMR-driven chain-of-thought prompting method, which we call AMRCoT.
We find that it is difficult to predict which input examples AMR may help or hurt on, but errors tend to arise with multi-word expressions.
arXiv Detail & Related papers (2024-05-02T17:32:59Z) - Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Models [14.906150451947443]
We propose a benchmark for evaluating long-context Large Language Models (LLMs) named Counting-Stars.
We conduct experiments to evaluate long-context LLMs (i.e., GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4, and Moonshot-v1)
Results show that Gemini 1.5 Pro achieves the best overall results, while GPT-4 Turbo is the most stable across various tasks.
arXiv Detail & Related papers (2024-03-18T14:01:45Z) - LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language
Models [56.25156596019168]
This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs)
Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
arXiv Detail & Related papers (2023-11-30T03:59:31Z) - LooGLE: Can Long-Context Language Models Understand Long Contexts? [50.408957515411096]
LooGLE is a benchmark for large language models' long context understanding.
It features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains.
The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings.
arXiv Detail & Related papers (2023-11-08T01:45:37Z) - LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding [58.20031627237889]
LongBench is the first bilingual, multi-task benchmark for long context understanding.
It comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese)
arXiv Detail & Related papers (2023-08-28T11:53:40Z) - Analysing the Effect of Masking Length Distribution of MLM: An
Evaluation Framework and Case Study on Chinese MRC Datasets [0.8566457170664925]
Masked language model (MLM) is a self-trained training objective widely used in various PTMs.
In different machine reading comprehension tasks, the length of the answer is also different, and the answer is often a word, phrase, or sentence.
In this paper, we try to uncover how much of four's success in the machine reading comprehension tasks comes from the correlation between masking length distribution and answer length in MRC dataset.
arXiv Detail & Related papers (2021-09-29T04:07:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.