TeXBLEU: Automatic Metric for Evaluate LaTeX Format

• URL: http://arxiv.org/abs/2409.06639v3
• Date: Fri, 13 Sep 2024 04:22:56 GMT
• Title: TeXBLEU: Automatic Metric for Evaluate LaTeX Format
• Authors: Kyudan Jung, Nam-Joon Kim, Hyongon Ryu, Sieun Hyeon, Seung-jun Lee, Hyeok-jae Lee,
• Abstract summary: We propose BLEU, a metric for evaluating mathematical expressions in the format built on the n-gram-based BLEU metric. The proposed BLEU consists of a tokenizer trained on the arXiv paper dataset and a fine-tuned embedding model with positional encoding.
• Score: 4.337656290539519
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LaTeX is suitable for creating specially formatted documents in science, technology, mathematics, and computer science. Although the use of mathematical expressions in LaTeX format along with language models is increasing, there are no proper evaluation matrices to evaluate them. In this study, we propose TeXBLEU, a metric for evaluating mathematical expressions in the LaTeX format built on the n-gram-based BLEU metric widely used in translation tasks. The proposed TeXBLEU consists of a predefined tokenizer trained on the arXiv paper dataset and a fine-tuned embedding model with positional encoding. The TeXBLEU score was calculated by replacing BLUE's modified precision score with the similarity of n-gram-based tokens. TeXBLEU showed improvements of 86\%, 121\%, and 610\% over traditional evaluation metrics, such as BLEU, sacreBLEU, and Rouge, respectively, on the MathBridge dataset with 1,000 data points. The code is available at https://github.com/KyuDan1/TeXBLEU.

Related papers

• LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement [11.931911831112357]
  LATTE improves the source extraction accuracy of both formulae and tables, outperforming existing techniques as well as GPT-4V. This paper proposes LATTE, the first iterative refinement framework for recognition.
  arXiv  Detail & Related papers  (2024-09-21T17:18:49Z)
• InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning [58.7966588457529]
  InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents. It comprises 24 million web pages, 85 million associated image URLs, and 40 billion text tokens, all meticulously extracted and filtered from CommonCrawl. Our evaluations on text-only benchmarks show that, despite utilizing only 40 billion tokens, our dataset significantly enhances the performance of our 1.3B model. Our models set a new state-of-the-art among open-source models on multi-modal math benchmarks such as MathVerse and We-Math.
  arXiv  Detail & Related papers  (2024-09-19T08:41:21Z)
• MathBridge: A Large Corpus Dataset for Translating Spoken Mathematical Expressions into $LaTeX$ Formulas for Improved Readability [10.757551947236879]
  We introduce MathBridge, the first extensive dataset for translating mathematical spoken sentences into formulas. MathBridge significantly enhances the capabilities of pretrained language models for converting to formulas from mathematical spoken sentences.
  arXiv  Detail & Related papers  (2024-08-07T18:07:15Z)
• MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition [2.325171167252542]
  We present an improved version of the benchmark dataset im2latex-100k, featuring 30 fonts instead of one. Second, we introduce the real-world dataset realFormula, with MEs extracted from papers. Third, we developed a MER model, MathNet, based on a convolutional vision transformer, with superior results on all four test sets.
  arXiv  Detail & Related papers  (2024-04-21T14:03:34Z)
• MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning [52.97768001837269]
  We present a method to fine-tune open-source language models, enabling them to use code for modeling and deriving math equations. We propose a method of generating novel and high-quality datasets with math problems and their code-based solutions. This approach yields the MathCoder models, a family of models capable of generating code-based solutions for solving challenging math problems.
  arXiv  Detail & Related papers  (2023-10-05T17:52:09Z)
• On the Blind Spots of Model-Based Evaluation Metrics for Text Generation [79.01422521024834]
  We explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics. We design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics.
  arXiv  Detail & Related papers  (2022-12-20T06:24:25Z)
• Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning [150.17907456113537]
  We present Tabular Math Word Problems (TabMWP), a new dataset containing 38,431 grade-level problems that require mathematical reasoning. We evaluate different pre-trained models on TabMWP, including the GPT-3 model in a few-shot setting. We propose a novel approach, PromptPG, which utilizes policy gradient to learn to select in-context examples from a small amount of training data.
  arXiv  Detail & Related papers  (2022-09-29T08:01:04Z)
• Syntax-Aware Network for Handwritten Mathematical Expression Recognition [53.130826547287626]
  Handwritten mathematical expression recognition (HMER) is a challenging task that has many potential applications. Recent methods for HMER have achieved outstanding performance with an encoder-decoder architecture. We propose a simple and efficient method for HMER, which is the first to incorporate syntax information into an encoder-decoder network.
  arXiv  Detail & Related papers  (2022-03-03T09:57:19Z)
• BARTScore: Evaluating Generated Text as Text Generation [89.50052670307434]
  We conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. We operationalize this idea using BART, an encoder-decoder based pre-trained model. We propose a metric BARTScore with a number of variants that can be flexibly applied to evaluation of text from different perspectives.
  arXiv  Detail & Related papers  (2021-06-22T03:20:53Z)
• Machine Translation of Mathematical Text [0.0]
  We have implemented a machine translation system, the PolyMath Translator, for documents containing mathematical text. The current implementation translates English to French, attaining a BLEU score of 53.5 on a held-out test corpus of mathematical sentences. It produces documents that can be compiled to PDF without further editing.
  arXiv  Detail & Related papers  (2020-10-11T11:59:40Z)
• Reproducible Science with LaTeX [4.09920839425892]
  This paper proposes a procedure to execute external source codes from a document. It includes the calculation outputs in the resulting Portable Document Format (pdf) file automatically.
  arXiv  Detail & Related papers  (2020-10-04T04:04:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.