Semantically Cohesive Word Grouping in Indian Languages

• URL: http://arxiv.org/abs/2501.03988v1
• Date: Tue, 07 Jan 2025 18:46:17 GMT
• Title: Semantically Cohesive Word Grouping in Indian Languages
• Authors: N J Karthika, Adyasha Patra, Nagasai Saketh Naidu, Arnab Bhattacharya, Ganesh Ramakrishnan, Chaitali Dangarikar,
• Abstract summary: Indian languages are inflectional and agglutinative and typically follow clause-free word order. The structure of sentences across most major Indian languages are similar when their dependency parse trees are considered. We propose word grouping as a major preprocessing step for any computational or linguistic processing of sentences for Indian languages.
• Score: 18.524219186230713
• License:
• Abstract: Indian languages are inflectional and agglutinative and typically follow clause-free word order. The structure of sentences across most major Indian languages are similar when their dependency parse trees are considered. While some differences in the parsing structure occur due to peculiarities of a language or its preferred natural way of conveying meaning, several apparent differences are simply due to the granularity of representation of the smallest semantic unit of processing in a sentence. The semantic unit is typically a word, typographically separated by whitespaces. A single whitespace-separated word in one language may correspond to a group of words in another. Hence, grouping of words based on semantics helps unify the parsing structure of parallel sentences across languages and, in the process, morphology. In this work, we propose word grouping as a major preprocessing step for any computational or linguistic processing of sentences for Indian languages. Among Indian languages, since Hindi is one of the least agglutinative, we expect it to benefit the most from word-grouping. Hence, in this paper, we focus on Hindi to study the effects of grouping. We perform quantitative assessment of our proposal with an intrinsic method that perturbs sentences by shuffling words as well as an extrinsic evaluation that verifies the importance of word grouping for the task of Machine Translation (MT) using decomposed prompting. We also qualitatively analyze certain aspects of the syntactic structure of sentences. Our experiments and analyses show that the proposed grouping technique brings uniformity in the syntactic structures, as well as aids underlying NLP tasks.

Related papers

• Urdu Dependency Parsing and Treebank Development: A Syntactic and Morphological Perspective [0.0]
  We use dependency parsing to analyze news articles in Urdu. We achieve a best-labeled accuracy (LA) of 70% and an unlabeled attachment score (UAS) of 84%.
  arXiv  Detail & Related papers  (2024-06-13T19:30:32Z)
• Syntactic Variation Across the Grammar: Modelling a Complex Adaptive System [0.76146285961466]
  We model dialectal variation across 49 local populations of English speakers in 16 countries. Results show that an important part of syntactic variation consists of interactions between different parts of the grammar. New Zealand English could be more similar to Australian English in phrasal verbs but at the same time more similar to UK English in dative phrases.
  arXiv  Detail & Related papers  (2023-09-21T08:14:34Z)
• PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition [63.51569687229681]
  We argue for the need to recognize the textual entailment relation of each proposition in a sentence individually. We propose PropSegmEnt, a corpus of over 45K propositions annotated by expert human raters. Our dataset structure resembles the tasks of (1) segmenting sentences within a document to the set of propositions, and (2) classifying the entailment relation of each proposition with respect to a different yet topically-aligned document.
  arXiv  Detail & Related papers  (2022-12-21T04:03:33Z)
• Discourse Context Predictability Effects in Hindi Word Order [14.88833412862455]
  We investigate how the words and syntactic structures in a sentence influence the word order of the following sentences. We use a number of discourse-based features and cognitive features to make its predictions, including dependency length, surprisal, and information status. We find that information status and LSTM-based discourse predictability influence word order choices, especially for non-canonical object-fronted orders.
  arXiv  Detail & Related papers  (2022-10-25T11:53:01Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• Quantifying Synthesis and Fusion and their Impact on Machine Translation [79.61874492642691]
  In Natural Language Processing (NLP) typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the rigidity of such claims, by quantifying morphological typology at the word and segment level. For computing literature, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study. Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish,
  arXiv  Detail & Related papers  (2022-05-06T17:04:58Z)
• Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences [69.3939291118954]
  This paper reports research on a set of comprehensive clustering and network analyses targeting sentence and sub-sentence embedding spaces. Results show that one method generates the most clusterable embeddings. In general, the embeddings of span sub-sentences have better clustering properties than the original sentences.
  arXiv  Detail & Related papers  (2021-10-02T00:47:35Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• Comparative Analysis of Word Embeddings for Capturing Word Similarities [0.0]
  Distributed language representation has become the most widely used technique for language representation in various natural language processing tasks. Most of the natural language processing models that are based on deep learning techniques use already pre-trained distributed word representations, commonly called word embeddings. selecting the appropriate word embeddings is a perplexing task since the projected embedding space is not intuitive to humans.
  arXiv  Detail & Related papers  (2020-05-08T01:16:03Z)
• On the Language Neutrality of Pre-trained Multilingual Representations [70.93503607755055]
  We investigate the language-neutrality of multilingual contextual embeddings directly and with respect to lexical semantics. Our results show that contextual embeddings are more language-neutral and, in general, more informative than aligned static word-type embeddings. We show how to reach state-of-the-art accuracy on language identification and match the performance of statistical methods for word alignment of parallel sentences.
  arXiv  Detail & Related papers  (2020-04-09T19:50:32Z)
• Unsupervised Separation of Native and Loanwords for Malayalam and Telugu [3.4925763160992402]
  Words from one language are adopted within a different language without translation; these words appear in transliterated form in text written in the latter language. This phenomenon is particularly widespread within Indian languages where many words are loaned from English. We address the task of identifying loanwords automatically and in an unsupervised manner, from large datasets of words from agglutinative Dravidian languages.
  arXiv  Detail & Related papers  (2020-02-12T04:01:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.