Language Complexity Measurement as a Noisy Zero-Shot Proxy for Evaluating LLM Performance

• URL: http://arxiv.org/abs/2502.11578v1
• Date: Mon, 17 Feb 2025 09:09:58 GMT
• Title: Language Complexity Measurement as a Noisy Zero-Shot Proxy for Evaluating LLM Performance
• Authors: Birger Moell, Johan Boye,
• Abstract summary: This paper investigates the performance of large language models (LLMs) on language complexity measurement tasks. Using Swedish high school and university-level essays, we evaluate the models' abilities to compute LIX scores and perform dependency parsing.
• Score: 1.6114012813668932
• License:
• Abstract: Large Language Models (LLMs) have made significant strides in natural language generation but often face challenges in tasks requiring precise calculations and structural analysis. This paper investigates the performance of state-of-the-art LLMs on language complexity measurement tasks, through the computation of the LIX readability metric and Average Dependency Distance (ADD). Using Swedish high school and university-level essays, we evaluate the models' abilities to compute LIX scores and perform dependency parsing, comparing their results to established ground truths. Our findings reveal that while all models demonstrate some capacity for these tasks, ChatGPT-o1-mini performs most consistently, achieving the highest accuracy in both LIX computation and dependency parsing. Additionally, we observe a strong significant correlation -0.875 p 0.026 (N=6) between the models' accuracy in computing LIX and their overall performance on the Massive Multitask Language Understanding (MMLU) benchmark. These results suggest that language complexity measurement abilities can serve as a noisy zero-shot proxies for assessing the general capabilities of LLMs, providing a practical method for model evaluation without the need for extensive benchmarking datasets.

Related papers

• Examining the Robustness of Large Language Models across Language Complexity [19.184633713069353]
  Large language models (LLMs) analyze textual artifacts generated by students to understand and evaluate their learning. This study examines the robustness of several LLM-based student models that detect student self-regulated learning (SRL) in math problem-solving.
  arXiv  Detail & Related papers  (2025-01-30T20:33:59Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models [9.710960283117771]
  ProxyLM is a task- and language-agnostic framework designed to predict the performance of LMs using proxy models. ProxyLM significantly reduces computational overhead in task evaluations, achieving up to a 37.08x speedup over traditional methods. Our results demonstrate that ProxyLM not only adapts well to previously unseen languages in pre-trained LMs, but also generalizes effectively across different datasets.
  arXiv  Detail & Related papers  (2024-06-13T17:15:33Z)
• TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale [66.01943465390548]
  We introduce TriSum, a framework for distilling large language models' text summarization abilities into a compact, local model. Our method enhances local model performance on various benchmarks. It also improves interpretability by providing insights into the summarization rationale.
  arXiv  Detail & Related papers  (2024-03-15T14:36:38Z)
• PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion [96.47420221442397]
  We construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. We test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates robustness settings. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark.
  arXiv  Detail & Related papers  (2024-03-06T15:33:32Z)
• Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization [132.25202059478065]
  We benchmark large language models (LLMs) on instruction controllable text summarization. Our study reveals that instruction controllable text summarization remains a challenging task for LLMs.
  arXiv  Detail & Related papers  (2023-11-15T18:25:26Z)
• Exploring the Potential of Large Language Models in Computational Argumentation [54.85665903448207]
  Large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language. This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2023-11-15T15:12:15Z)
• Assessing the Reliability of Large Language Model Knowledge [78.38870272050106]
  Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks. How do we evaluate the capabilities of LLMs to consistently produce factually correct answers? We propose MOdel kNowledge relIabiliTy scORe (MONITOR), a novel metric designed to directly measure LLMs' factual reliability.
  arXiv  Detail & Related papers  (2023-10-15T12:40:30Z)
• Benchmarking Causal Study to Interpret Large Language Models for Source Code [6.301373791541809]
  This paper introduces a benchmarking strategy named Galeras comprised of curated testbeds for three SE tasks. We illustrate the insights of our benchmarking strategy by conducting a case study on the performance of ChatGPT under distinct prompt engineering methods.
  arXiv  Detail & Related papers  (2023-08-23T20:32:12Z)
• RoCar: A Relationship Network-based Evaluation Method for Large Language Models [20.954826722195847]
  How to rationally evaluate the capabilities of large language models (LLMs) is still a task to be solved. We propose the RoCar method, which utilizes the defined basic schemas to randomly construct a task graph. It is possible to ensure that none of the LLMs to be tested has directly learned the evaluation tasks, guaranteeing the fairness of the evaluation method.
  arXiv  Detail & Related papers  (2023-07-29T14:47:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.