An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses

• URL: http://arxiv.org/abs/2503.21393v2
• Date: Wed, 02 Apr 2025 03:17:30 GMT
• Title: An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses
• Authors: Rohitash Chandra, Aryan Chaudhari, Yeshwanth Rayavarapu,
• Abstract summary: Large Language models (LLMs) have been prominent for language translation, including low-resource languages.<n>This study uses semantic and sentiment analysis of selected LLMs for Indian languages, including Sanskrit, Telugu and Hindi.
• Score: 0.17999333451993949
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language models (LLMs) have been prominent for language translation, including low-resource languages. There has been limited study about the assessment of the quality of translations generated by LLMs, including Gemini, GPT and Google Translate. In this study, we address this limitation by using semantic and sentiment analysis of selected LLMs for Indian languages, including Sanskrit, Telugu and Hindi. We select prominent texts that have been well translated by experts and use LLMs to generate their translations to English, and then we provide a comparison with selected expert (human) translations. Our findings suggest that while LLMs have made significant progress in translation accuracy, challenges remain in preserving sentiment and semantic integrity, especially in figurative and philosophical contexts. The sentiment analysis revealed that GPT-4o and GPT-3.5 are better at preserving the sentiments for the Bhagavad Gita (Sanskrit-English) translations when compared to Google Translate. We observed a similar trend for the case of Tamas (Hindi-English) and Maha P (Telugu-English) translations. GPT-4o performs similarly to GPT-3.5 in the translation in terms of sentiments for the three languages. We found that LLMs are generally better at translation for capturing sentiments when compared to Google Translate.

Related papers

• Lost in Literalism: How Supervised Training Shapes Translationese in LLMs [51.04435855143767]
  Large language models (LLMs) have achieved remarkable success in machine translation.<n>However, translationese, characterized by overly literal and unnatural translations, remains a persistent challenge.<n>We introduce methods to mitigate these biases, including polishing golden references and filtering unnatural training instances.
  arXiv  Detail & Related papers  (2025-03-06T12:14:45Z)
• Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels [20.05501751993599]
  GPT-4 achieves performance comparable to junior-level translators in terms of total errors. Unlike traditional Neural Machine Translation systems, GPT-4 maintains consistent translation quality across all evaluated language pairs.
  arXiv  Detail & Related papers  (2024-11-21T01:12:46Z)
• Evaluation of Google Translate for Mandarin Chinese translation using sentiment and semantic analysis [1.3999481573773074]
  Machine translation using large language models (LLMs) is having a significant global impact. Mandarin Chinese is the official language used for communication by the government and media in China. In this study, we provide an automated assessment of translation quality of Google Translate with human experts using sentiment and semantic analysis.
  arXiv  Detail & Related papers  (2024-09-08T04:03:55Z)
• GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels [18.835573312027265]
  This study comprehensively evaluates the translation quality of Large Language Models (LLMs) against human translators. We find that GPT-4 performs comparably to junior translators in terms of total errors made but lags behind medium and senior translators.
  arXiv  Detail & Related papers  (2024-07-04T05:58:04Z)
• (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts [56.7988577327046]
  We introduce TransAgents, a novel multi-agent framework that simulates the roles and collaborative practices of a human translation company. Our findings highlight the potential of multi-agent collaboration in enhancing translation quality, particularly for longer texts.
  arXiv  Detail & Related papers  (2024-05-20T05:55:08Z)
• Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning [57.323716555996114]
  Off-target translation remains an unsolved problem, especially for low-resource languages. Recent works have either designed advanced prompting strategies to highlight the functionality of translation instructions or exploited the in-context learning ability of LLMs. In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability (especially the translation direction) of LLMs.
  arXiv  Detail & Related papers  (2024-03-21T13:47:40Z)
• Do GPTs Produce Less Literal Translations? [20.095646048167612]
  Large Language Models (LLMs) have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. We find that translations out of English (E-X) from GPTs tend to be less literal, while exhibiting similar or better scores on Machine Translation quality metrics.
  arXiv  Detail & Related papers  (2023-05-26T10:38:31Z)
• Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions [68.01449013641532]
  Large-scale Pretrained Language Models (LLMs) have shown strong abilities in multilingual translations. We present a detailed analysis by finetuning a multilingual pretrained language model, XGLM-7B, to perform multilingual translation.
  arXiv  Detail & Related papers  (2023-05-24T12:00:24Z)
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis [103.89753784762445]
  Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT) This paper systematically investigates the advantages and challenges of LLMs for MMT. We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4.
  arXiv  Detail & Related papers  (2023-04-10T15:51:30Z)
• Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine [97.8609714773255]
  We evaluate ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. ChatGPT performs competitively with commercial translation products but lags behind significantly on low-resource or distant languages. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted.
  arXiv  Detail & Related papers  (2023-01-20T08:51:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.