Investigating Numerical Translation with Large Language Models

• URL: http://arxiv.org/abs/2501.04927v1
• Date: Thu, 09 Jan 2025 02:32:40 GMT
• Title: Investigating Numerical Translation with Large Language Models
• Authors: Wei Tang, Jiawei Yu, Yuang Li, Yanqing Zhao, Weidong Zhang, Wei Feng, Min Zhang, Hao Yang,
• Abstract summary: This study focuses on evaluating the reliability of large language models (LLMs) when handling numerical data. Experiments on the dataset indicate that errors in numerical translation are a common issue. Even the latest llama3.1 8b model can have error rates as high as 20%.
• Score: 27.016480167041905
• License:
• Abstract: The inaccurate translation of numbers can lead to significant security issues, ranging from financial setbacks to medical inaccuracies. While large language models (LLMs) have made significant advancements in machine translation, their capacity for translating numbers has not been thoroughly explored. This study focuses on evaluating the reliability of LLM-based machine translation systems when handling numerical data. In order to systematically test the numerical translation capabilities of currently open source LLMs, we have constructed a numerical translation dataset between Chinese and English based on real business data, encompassing ten types of numerical translation. Experiments on the dataset indicate that errors in numerical translation are a common issue, with most open-source LLMs faltering when faced with our test scenarios. Especially when it comes to numerical types involving large units like ``million", ``billion", and "yi", even the latest llama3.1 8b model can have error rates as high as 20%. Finally, we introduce three potential strategies to mitigate the numerical mistranslations for large units.

Related papers

• Exploring Large Language Models for Translating Romanian Computational Problems into English [0.0]
  This study shows that robust large language models (LLMs) can maintain or even enhance their performance in translating less common languages when given well-structured prompts. We evaluate several translation methods across multiple LLMs, including OpenRoLLM, Llama 3.1 8B, Llama 3.2 3B and GPT-4o.
  arXiv  Detail & Related papers  (2025-01-09T22:17:44Z)
• What do Large Language Models Need for Machine Translation Evaluation? [12.42394213466485]
  Large language models (LLMs) can achieve results comparable to fine-tuned multilingual pre-trained language models. This paper explores what translation information, such as the source, reference, translation errors and annotation guidelines, is needed for LLMs to evaluate machine translation quality.
  arXiv  Detail & Related papers  (2024-10-04T09:50:45Z)
• Advancing Anomaly Detection: Non-Semantic Financial Data Encoding with LLMs [49.57641083688934]
  We introduce a novel approach to anomaly detection in financial data using Large Language Models (LLMs) embeddings. Our experiments demonstrate that LLMs contribute valuable information to anomaly detection as our models outperform the baselines.
  arXiv  Detail & Related papers  (2024-06-05T20:19:09Z)
• Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis [3.16714407449467]
  We investigate the role of translation and synthetic data in training language models. We translate TinyStories, a dataset of 2.2M short stories for 3-4 year old children, from English to Arabic using the open NLLB-3B MT model. To rectify these issues, we pre-train the models with a small dataset of synthesized high-quality Arabic stories.
  arXiv  Detail & Related papers  (2024-05-23T07:53:04Z)
• Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice? [33.376648335299116]
  Large language models (LLMs) display strong translation capability after being fine-tuned on as few as 32 parallel sentences. LLMs with only English on the target side can lead to task misinterpretation, which hinders translation into non-English languages. synthesized data in an under-represented language has a less pronounced effect.
  arXiv  Detail & Related papers  (2024-04-22T12:21:12Z)
• TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinement [26.26493253161022]
  Large Language Models (LLMs) have achieved impressive results in Machine Translation (MT) We introduce a systematic LLM-based self-refinement translation framework, named textbfTEaR.
  arXiv  Detail & Related papers  (2024-02-26T07:58:12Z)
• Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation [64.5862977630713]
  This study investigates how Large Language Models (LLMs) leverage source and reference data in machine translation evaluation task. We find that reference information significantly enhances the evaluation accuracy, while surprisingly, source information sometimes is counterproductive.
  arXiv  Detail & Related papers  (2024-01-12T13:23:21Z)
• Chain-of-Dictionary Prompting Elicits Translation in Large Language Models [100.47154959254937]
  Large language models (LLMs) have shown surprisingly good performance in multilingual neural machine translation (MNMT) We present a novel method, CoD, which augments LLMs with prior knowledge with the chains of multilingual dictionaries for a subset of input words to elicit translation abilities.
  arXiv  Detail & Related papers  (2023-05-11T05:19:47Z)
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis [103.89753784762445]
  Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT) This paper systematically investigates the advantages and challenges of LLMs for MMT. We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4.
  arXiv  Detail & Related papers  (2023-04-10T15:51:30Z)
• As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation [51.20569527047729]
  Mistranslated numbers have the potential to cause serious effects, such as financial loss or medical misinformation. We develop comprehensive assessments of the robustness of neural machine translation systems to numerical text via behavioural testing.
  arXiv  Detail & Related papers  (2021-07-18T04:09:47Z)
• Beyond English-Centric Multilingual Machine Translation [74.21727842163068]
  We create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT.
  arXiv  Detail & Related papers  (2020-10-21T17:01:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.