Mitrasamgraha: A Comprehensive Classical Sanskrit Machine Translation Dataset

• URL: http://arxiv.org/abs/2601.07314v1
• Date: Mon, 12 Jan 2026 08:37:15 GMT
• Title: Mitrasamgraha: A Comprehensive Classical Sanskrit Machine Translation Dataset
• Abstract summary: Mitrasamgraha is a high-quality Sanskrit-to-English machine translation dataset consisting of 391,548 bitext pairs.<n>It covers a time period of more than three millennia and a broad range of historical Sanskrit domains.<n>We conduct experiments benchmarking commercial and open models on this dataset and fine-tune NLLB and Gemma models on the dataset.
• Score: 26.899919193282944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While machine translation is regarded as a "solved problem" for many high-resource languages, close analysis quickly reveals that this is not the case for content that shows challenges such as poetic language, philosophical concepts, multi-layered metaphorical expressions, and more. Sanskrit literature is a prime example of this, as it combines a large number of such challenges in addition to inherent linguistic features like sandhi, compounding, and heavy morphology, which further complicate NLP downstream tasks. It spans multiple millennia of text production time as well as a large breadth of different domains, ranging from ritual formulas via epic narratives, philosophical treatises, poetic verses up to scientific material. As of now, there is a strong lack of publicly available resources that cover these different domains and temporal layers of Sanskrit. We therefore introduce Mitrasamgraha, a high-quality Sanskrit-to-English machine translation dataset consisting of 391,548 bitext pairs, more than four times larger than the largest previously available Sanskrit dataset Itih=asa. It covers a time period of more than three millennia and a broad range of historical Sanskrit domains. In contrast to web-crawled datasets, the temporal and domain annotation of this dataset enables fine-grained study of domain and time period effects on MT performance. We also release a validation set consisting of 5,587 and a test set consisting of 5,552 post-corrected bitext pairs. We conduct experiments benchmarking commercial and open models on this dataset and fine-tune NLLB and Gemma models on the dataset, showing significant improvements, while still recognizing significant challenges in the translation of complex compounds, philosophical concepts, and multi-layered metaphors. We also analyze how in-context learning on this dataset impacts the performance of commercial models

Related papers

• MameLoshnLM: Yiddish Language Model and Evaluation Benchmark [62.936233688546984]
  We present MameLoshnLM, the first open-source 8B- parameter language model built specifically for Yiddish.<n>Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling.<n>Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.
  arXiv  Detail & Related papers  (2026-08-06T10:24:08Z)
• Samasāmayik: A Parallel Dataset for Hindi-Sanskrit Machine Translation [12.522148604600341]
  Samasmayik is a novel, meticulously curated, large-scale Hindi-Sanskrit corpus, comprising 92,196 parallel sentences.<n>We benchmark this new dataset by fine-tuning three complementary models - ByT5, NLLB and IndicTrans-v2 to demonstrate its utility.<n>Our experiments demonstrate that models trained on the Samasamayik corpus achieve significant performance gains on in-domain test data, while achieving comparable performance on other widely used test sets.
  arXiv  Detail & Related papers  (2026-03-25T13:48:34Z)
• Mahānāma: A Unique Testbed for Literary Entity Discovery and Linking [5.247057581790816]
  Mah=an=ama is the first large-scale dataset for end-to-end Entity Discovery and Linking in Sanskrit.<n>The dataset comprises over 109K named entity mentions mapped to 5.5K unique entities.<n>The complex narrative structure of Mah=an=ama, coupled with extensive name variation and ambiguity, poses significant challenges to resolution systems.
  arXiv  Detail & Related papers  (2025-09-24T07:42:39Z)
• TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation [88.14715494221123]
  We introduce TextAtlas5M, a novel dataset designed to evaluate long-text rendering in text-conditioned image generation.<n>Our dataset consists of 5 million long-text generated and collected images across diverse data types.<n>We further curate 3000 human-improved test set TextAtlasEval across 3 data domains, establishing one of the most extensive benchmarks for text-conditioned generation.
  arXiv  Detail & Related papers  (2025-02-11T18:59:19Z)
• Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languages [27.273651323572786]
  BhasaAnuvaad is the largest speech translation dataset for Indian languages, spanning over 44 thousand hours of audio and 17 million aligned text segments.<n>Our experiments demonstrate improvements in the translation quality, setting a new standard for Indian language speech translation.<n>We will release all the code, data and model weights in the open-source, with permissive licenses to promote accessibility and collaboration.
  arXiv  Detail & Related papers  (2024-11-07T13:33:34Z)
• ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation [1.3750624267664155]
  Classical Arabic represents a significant era that encompasses the golden age of Arab culture, philosophy, and scientific literature.<n>We have identified a scarcity of translation datasets in Classical Arabic, which are often limited in scope and topics.<n>We present the ATHAR dataset, which comprises 66,000 high-quality classical Arabic to English translation samples.
  arXiv  Detail & Related papers  (2024-07-29T09:45:34Z)
• TRINS: Towards Multimodal Language Models that Can Read [61.17806538631744]
  TRINS is a Text-Rich image INStruction dataset. It contains 39,153 text-rich images, captions, and 102,437 questions. We introduce a Language-vision Reading Assistant (LaRA) which is good at understanding textual content within images.
  arXiv  Detail & Related papers  (2024-06-10T18:52:37Z)
• IndicSTR12: A Dataset for Indic Scene Text Recognition [33.194567434881314]
  This paper proposes the largest and most comprehensive real dataset - IndicSTR12 - and benchmarking STR performance on 12 major Indian languages. The size and complexity of the proposed dataset are comparable to those of existing Latin contemporaries. The dataset contains over 27000 word-images gathered from various natural scenes, with over 1000 word-images for each language.
  arXiv  Detail & Related papers  (2024-03-12T18:14:48Z)
• SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages [44.017657230247934]
  We present textitSemRel, a new semantic relatedness dataset collection annotated by native speakers across 13 languages. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences.
  arXiv  Detail & Related papers  (2024-02-13T18:04:53Z)
• Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models [91.6543868677356]
  The evolution of Neural Machine Translation has been influenced by six core challenges. These challenges include domain mismatch, amount of parallel data, rare word prediction, translation of long sentences, attention model as word alignment, and sub-optimal beam search. This study revisits these challenges, offering insights into their ongoing relevance in the context of advanced Large Language Models.
  arXiv  Detail & Related papers  (2024-01-16T13:30:09Z)
• What's In My Big Data? [67.04525616289949]
  We propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities -- count and search -- at scale, which allows us to analyze more than 35 terabytes on a standard compute node. Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content.
  arXiv  Detail & Related papers  (2023-10-31T17:59:38Z)
• Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems [64.40789703661987]
  Multi3WOZ is a novel multilingual, multi-domain, multi-parallel ToD dataset. It is large-scale and offers culturally adapted dialogs in 4 languages. We describe a complex bottom-up data collection process that yielded the final dataset.
  arXiv  Detail & Related papers  (2023-07-26T08:29:42Z)
• Sāmayik: A Benchmark and Dataset for English-Sanskrit Translation [30.315293326789828]
  S=amayik is a dataset of around 53,000 parallel English-Sanskrit sentences, written in contemporary prose. S=amayik is curated from a diverse range of domains, including language instruction material, textual teaching pedagogy, and online tutorials.
  arXiv  Detail & Related papers  (2023-05-23T12:32:24Z)
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models [59.107260266206445]
  Short texts published on Twitter have earned significant attention as a rich source of information. Their inherent characteristics, such as the informal, and noisy linguistic style, remain challenging to many natural language processing (NLP) tasks. This study fulfils an assessment of existing language models in distinguishing the sentiment expressed in tweets by using a rich collection of 22 datasets.
  arXiv  Detail & Related papers  (2021-05-29T21:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.