TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs

• URL: http://arxiv.org/abs/2506.16990v1
• Date: Fri, 20 Jun 2025 13:39:16 GMT
• Title: TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs
• Authors: Sahil Kale, Vijaykant Nadadur,
• Abstract summary: Large Language Models (LLMs) present a promising opportunity for researchers to produce publication-ready material.<n>Our benchmark dataset with natural language prompts for generating code focused on components of scientific documents.<n>Our evaluation across open and closed-source LLMs highlights multiple key findings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LaTeX's precision and flexibility in typesetting have made it the gold standard for the preparation of scientific documentation. Large Language Models (LLMs) present a promising opportunity for researchers to produce publication-ready material using LaTeX with natural language instructions, yet current benchmarks completely lack evaluation of this ability. By introducing TeXpert, our benchmark dataset with natural language prompts for generating LaTeX code focused on components of scientific documents across multiple difficulty levels, we conduct an in-depth analysis of LLM performance in this regard and identify frequent error types. Our evaluation across open and closed-source LLMs highlights multiple key findings: LLMs excelling on standard benchmarks perform poorly in LaTeX generation with a significant accuracy drop-off as the complexity of tasks increases; open-source models like DeepSeek v3 and DeepSeek Coder strongly rival closed-source counterparts in LaTeX tasks; and formatting and package errors are unexpectedly prevalent, suggesting a lack of diverse LaTeX examples in the training datasets of most LLMs. Our dataset, code, and model evaluations are available at https://github.com/knowledge-verse-ai/TeXpert.

Related papers

• NeuRaLaTeX: A machine learning library written in pure LaTeX [15.978130916451295]
  We introduce NeuRaLa, which we believe to be the first deep learning library written entirely in rhyme.<n>As part of your document you can specify the architecture of a neural network and its loss functions.<n>When the document is compiled, the compiler will generate or load training data, train the network, run experiments, and generate figures.<n>The paper took 48 hours to compile and the entire source code for NeuRaLa is contained within the source code of the paper.
  arXiv  Detail & Related papers  (2025-03-31T15:05:19Z)
• LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement [11.931911831112357]
  The source and rendered PDF images look drastically different, especially for formulae and tables.<n>Prior work generates sources in a single iteration and struggles with complex formulae.<n>This paper proposes LATTE, the first iterative refinement framework for recognition.
  arXiv  Detail & Related papers  (2024-09-21T17:18:49Z)
• TeXBLEU: Automatic Metric for Evaluate LaTeX Format [4.337656290539519]
  We propose BLEU, a metric for evaluating mathematical expressions in the format built on the n-gram-based BLEU metric. The proposed BLEU consists of a tokenizer trained on the arXiv paper dataset and a fine-tuned embedding model with positional encoding.
  arXiv  Detail & Related papers  (2024-09-10T16:54:32Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition [2.325171167252542]
  We present an improved version of the benchmark dataset im2latex-100k, featuring 30 fonts instead of one. Second, we introduce the real-world dataset realFormula, with MEs extracted from papers. Third, we developed a MER model, MathNet, based on a convolutional vision transformer, with superior results on all four test sets.
  arXiv  Detail & Related papers  (2024-04-21T14:03:34Z)
• Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks [76.43527940649939]
  We introduce Ada-LEval, a benchmark for evaluating the long-context understanding of large language models (LLMs) Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable a more reliable evaluation of LLMs' long context capabilities. We evaluate 4 state-of-the-art closed-source API models and 6 open-source models with Ada-LEval.
  arXiv  Detail & Related papers  (2024-04-09T17:30:48Z)
• CodecLM: Aligning Language Models with Tailored Synthetic Data [51.59223474427153]
  We introduce CodecLM, a framework for adaptively generating high-quality synthetic data for instruction-following abilities. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples.
  arXiv  Detail & Related papers  (2024-04-08T21:15:36Z)
• PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion [96.47420221442397]
  We construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. We test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates robustness settings. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark.
  arXiv  Detail & Related papers  (2024-03-06T15:33:32Z)
• mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model [73.38800189095173]
  This work focuses on strengthening the multi-modal diagram analysis ability of Multimodal LLMs. By parsing Latex source files of high-quality papers, we carefully build a multi-modal diagram understanding dataset M-Paper. M-Paper is the first dataset to support joint comprehension of multiple scientific diagrams, including figures and tables in the format of images or Latex codes.
  arXiv  Detail & Related papers  (2023-11-30T04:43:26Z)
• AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations [52.43593893122206]
  Alignedcot is an in-context learning technique for invoking Large Language Models. It achieves consistent and correct step-wise prompts in zero-shot scenarios. We conduct experiments on mathematical reasoning and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-11-22T17:24:21Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data? [49.688233418425995]
  Struc-Bench is a comprehensive benchmark featuring prominent Large Language Models (LLMs) We propose two innovative metrics, P-Score (Prompting Score) and H-Score (Heuristical Score) Our experiments show that applying our structure-aware fine-tuning to LLaMA-7B leads to substantial performance gains.
  arXiv  Detail & Related papers  (2023-09-16T11:31:58Z)
• Reproducible Science with LaTeX [4.09920839425892]
  This paper proposes a procedure to execute external source codes from a document. It includes the calculation outputs in the resulting Portable Document Format (pdf) file automatically.
  arXiv  Detail & Related papers  (2020-10-04T04:04:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.