Pralekha: An Indic Document Alignment Evaluation Benchmark

• URL: http://arxiv.org/abs/2411.19096v1
• Date: Thu, 28 Nov 2024 12:17:24 GMT
• Title: Pralekha: An Indic Document Alignment Evaluation Benchmark
• Authors: Sanjay Suryanarayanan, Haiyue Song, Mohammed Safi Ur Rahman Khan, Anoop Kunchukuttan, Mitesh M. Khapra, Raj Dabre,
• Abstract summary: We introduce Pralekha, a large-scale benchmark for document-level alignment evaluation.<n>Pralekha includes over 2 million documents, with a 1:2 ratio of unaligned to aligned pairs, covering 11 Indic languages and English.<n>We propose Document Alignment Coefficient to address the challenge of aligning documents using sentence and chunk-level alignments.
• Score: 34.05246795948451
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mining parallel document pairs poses a significant challenge because existing sentence embedding models often have limited context windows, preventing them from effectively capturing document-level information. Another overlooked issue is the lack of concrete evaluation benchmarks comprising high-quality parallel document pairs for assessing document-level mining approaches, particularly for Indic languages. In this study, we introduce Pralekha, a large-scale benchmark for document-level alignment evaluation. Pralekha includes over 2 million documents, with a 1:2 ratio of unaligned to aligned pairs, covering 11 Indic languages and English. Using Pralekha, we evaluate various document-level mining approaches across three dimensions: the embedding models, the granularity levels, and the alignment algorithm. To address the challenge of aligning documents using sentence and chunk-level alignments, we propose a novel scoring method, Document Alignment Coefficient (DAC). DAC demonstrates substantial improvements over baseline pooling approaches, particularly in noisy scenarios, achieving average gains of 20-30% in precision and 15-20% in F1 score. These results highlight DAC's effectiveness in parallel document mining for Indic languages.

Related papers

• Reference-Based Post-OCR Processing with LLM for Precise Diacritic Text in Historical Document Recognition [1.6941039309214678]
  We propose a method utilizing available content-focused ebooks as a reference base to correct imperfect OCR-generated text.<n>This technique generates high-precision pseudo-page-to-page labels for diacritic languages.<n>The pipeline eliminates various types of noise from aged documents and addresses issues such as missing characters, words, and disordered sequences.
  arXiv  Detail & Related papers  (2024-10-17T08:05:02Z)
• Contextual Document Embeddings [77.22328616983417]
  We propose two complementary methods for contextualized document embeddings. First, an alternative contrastive learning objective that explicitly incorporates the document neighbors into the intra-batch contextual loss. Second, a new contextual architecture that explicitly encodes neighbor document information into the encoded representation.
  arXiv  Detail & Related papers  (2024-10-03T14:33:34Z)
• The Power of Summary-Source Alignments [62.76959473193149]
  Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection. alignment of corresponding sentences between a reference summary and its source documents has been leveraged to generate training data. This paper proposes extending the summary-source alignment framework by applying it at the more fine-grained proposition span level.
  arXiv  Detail & Related papers  (2024-06-02T19:35:19Z)
• SentAlign: Accurate and Scalable Sentence Alignment [4.363828136730248]
  SentAlign is an accurate sentence alignment tool designed to handle very large parallel document pairs. The alignment algorithm evaluates all possible alignment paths in fairly large documents of thousands of sentences and uses a divide-and-conquer approach to align documents containing tens of thousands of sentences.
  arXiv  Detail & Related papers  (2023-11-15T14:15:41Z)
• In-context Pretraining: Language Modeling Beyond Document Boundaries [137.53145699439898]
  In-Context Pretraining is a new approach where language models are pretrained on a sequence of related documents. We introduce approximate algorithms for finding related documents with efficient nearest neighbor search. We see notable improvements in tasks that require more complex contextual reasoning.
  arXiv  Detail & Related papers  (2023-10-16T17:57:12Z)
• A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents [0.5812284760539713]
  We define this problem as "scarce annotated legal documents" We propose a deep-learning-based classification framework which we call MESc. We also propose an explanation extraction algorithm named ORSE.
  arXiv  Detail & Related papers  (2023-09-19T12:18:28Z)
• WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction [31.96433679860807]
  Most existing word alignment methods rely on manual alignment datasets or parallel corpora. We relax the requirement for correct, fully-aligned, and parallel sentences. We then use such a large-scale weakly-supervised dataset for word alignment pre-training via span prediction.
  arXiv  Detail & Related papers  (2023-06-09T03:11:42Z)
• DAPR: A Benchmark on Document-Aware Passage Retrieval [57.45793782107218]
  We propose and name this task emphDocument-Aware Passage Retrieval (DAPR) While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context. Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
  arXiv  Detail & Related papers  (2023-05-23T10:39:57Z)
• Dual-Alignment Pre-training for Cross-lingual Sentence Embedding [79.98111074307657]
  We propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding. We introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. Our approach can significantly improve sentence embedding.
  arXiv  Detail & Related papers  (2023-05-16T03:53:30Z)
• Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? [18.968571816913208]
  We provide a systematic comparison of methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. We show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit.
  arXiv  Detail & Related papers  (2023-04-28T12:11:21Z)
• Majority Voting with Bidirectional Pre-translation For Bitext Retrieval [2.580271290008534]
  A popular approach has been to mine so-called "pseudo-parallel" sentences from paired documents in two languages. In this paper, we outline some problems with current methods, propose computationally economical solutions to those problems, and demonstrate success with novel methods. We make the code and data used for our experiments publicly available.
  arXiv  Detail & Related papers  (2021-03-10T22:24:01Z)
• CDA: a Cost Efficient Content-based Multilingual Web Document Aligner [97.98885151955467]
  We introduce a Content-based Document Alignment approach to align multilingual web documents based on content. We leverage lexical translation models to build vector representations using TF-IDF. Experiments show that CDA is robust, cost-effective, and is significantly superior in (i) processing large and noisy web data and (ii) scaling to new and low-resourced languages.
  arXiv  Detail & Related papers  (2021-02-20T03:37:23Z)
• A Comparison of Approaches to Document-level Machine Translation [34.2276281264886]
  This paper presents a systematic comparison of selected approaches to document-level phenomena evaluation suites. We find that a simple method based purely on back-translating monolingual document-level data performs as well as much more elaborate alternatives.
  arXiv  Detail & Related papers  (2021-01-26T19:21:09Z)
• Multitask Learning for Class-Imbalanced Discourse Classification [74.41900374452472]
  We show that a multitask approach can improve 7% Micro F1-score upon current state-of-the-art benchmarks. We also offer a comparative review of additional techniques proposed to address resource-poor problems in NLP.
  arXiv  Detail & Related papers  (2021-01-02T07:13:41Z)
• Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
  We argue that previous research did not make a clear use of the global context. We propose a new document-level NMT framework that deliberately models the local context of each sentence.
  arXiv  Detail & Related papers  (2020-02-19T03:30:00Z)
• Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover's Distance [8.395430195053061]
  Document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. We develop an unsupervised scoring function that leverages cross-lingual sentence embeddings to compute the semantic distance between documents in different languages. These semantic distances are then used to guide a document alignment algorithm to properly pair cross-lingual web documents across a variety of low, mid, and high-resource language pairs.
  arXiv  Detail & Related papers  (2020-01-31T05:14:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.