Entropy and type-token ratio in gigaword corpora

• URL: http://arxiv.org/abs/2411.10227v2
• Date: Wed, 26 Feb 2025 22:21:27 GMT
• Title: Entropy and type-token ratio in gigaword corpora
• Authors: Pablo Rosillo-Rodes, Maxi San Miguel, David Sanchez,
• Abstract summary: lexical diversity is characterized in terms of the type-token ratio and the word entropy.<n>We investigate both diversity metrics in six massive linguistic datasets in English, Spanish, and Turkish.<n>We unveil an empirical functional relation between entropy and type-token ratio of texts of a given corpus and language.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There are different ways of measuring diversity in complex systems. In particular, in language, lexical diversity is characterized in terms of the type-token ratio and the word entropy. We here investigate both diversity metrics in six massive linguistic datasets in English, Spanish, and Turkish, consisting of books, news articles, and tweets. These gigaword corpora correspond to languages with distinct morphological features and differ in registers and genres, thus constituting a varied testbed for a quantitative approach to lexical diversity. We unveil an empirical functional relation between entropy and type-token ratio of texts of a given corpus and language, which is a consequence of the statistical laws observed in natural language. Further, in the limit of large text lengths we find an analytical expression for this relation relying on both Zipf and Heaps laws that agrees with our empirical findings.

Related papers

• Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent [22.63155507847401]
  We predict that languages that use prosody to make lexical distinctions should exhibit a higher mutual information between word identity and prosody, compared to languages that don't.<n>We use a dataset of speakers reading sentences aloud in ten languages across five language families to estimate the mutual information between the text and their pitch curves.
  arXiv  Detail & Related papers  (2025-05-12T15:25:17Z)
• Variationist: Exploring Multifaceted Variation and Bias in Written Language Data [3.666781404469562]
  Exploring and understanding language data is a fundamental stage in all areas dealing with human language. Yet, there is currently a lack of a unified, customizable tool to seamlessly inspect and visualize language variation and bias. In this paper, we introduce Variationist, a highly-modular, descriptive, and task-agnostic tool that fills this gap.
  arXiv  Detail & Related papers  (2024-06-25T15:41:07Z)
• Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models [0.0]
  This paper examines the impact of tokenization strategies and vocabulary sizes on the performance of Arabic language models. Our study uncovers limited impacts of vocabulary size on model performance while keeping the model size unchanged. Paper's recommendations include refining tokenization strategies to address dialect challenges, enhancing model robustness across diverse linguistic contexts, and expanding datasets to encompass the rich dialect based Arabic.
  arXiv  Detail & Related papers  (2024-03-17T07:44:44Z)
• Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores [28.431348662950743]
  We release Python package for measuring and extracting repetition in text. We build a platform based on diversity for users to interactively explore repetition in text.
  arXiv  Detail & Related papers  (2024-03-01T14:23:12Z)
• Syntactic Language Change in English and German: Metrics, Parsers, and Convergences [56.47832275431858]
  The current paper looks at diachronic trends in syntactic language change in both English and German, using corpora of parliamentary debates from the last c. 160 years. We base our observations on five dependencys, including the widely used Stanford Core as well as 4 newer alternatives. We show that changes in syntactic measures seem to be more frequent at the tails of sentence length distributions.
  arXiv  Detail & Related papers  (2024-02-18T11:46:16Z)
• Patterns of Persistence and Diffusibility across the World's Languages [3.7055269158186874]
  Colexification is a type of similarity where a single lexical form is used to convey multiple meanings. We shed light on the linguistic causes of cross-lingual similarity in colexification and phonology. We construct large-scale graphs incorporating semantic, genealogical, phonological and geographical data for 1,966 languages.
  arXiv  Detail & Related papers  (2024-01-03T12:05:38Z)
• Lexical Diversity in Kinship Across Languages and Dialects [6.80465507148218]
  We introduce a method to enrich computational lexicons with content relating to linguistic diversity. The method is verified through two large-scale case studies on kinship terminology.
  arXiv  Detail & Related papers  (2023-08-24T19:49:30Z)
• Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? [50.48082721476612]
  Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability. We investigate the distributions of grammatical relations induced from mBERT in the context of 24 typologically different languages.
  arXiv  Detail & Related papers  (2022-12-21T09:44:08Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Corpus-Guided Contrast Sets for Morphosyntactic Feature Detection in Low-Resource English Varieties [3.3536302616846734]
  We present a human-in-the-loop approach to generate and filter effective contrast sets via corpus-guided edits. We show that our approach improves feature detection for both Indian English and African American English, demonstrate how it can assist linguistic research, and release our fine-tuned models for use by other researchers.
  arXiv  Detail & Related papers  (2022-09-15T21:19:31Z)
• Universality and diversity in word patterns [0.0]
  We present an analysis of lexical statistical connections for eleven major languages. We find that the diverse manners that languages utilize to express word relations give rise to unique pattern distributions.
  arXiv  Detail & Related papers  (2022-08-23T20:03:27Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations [83.27475281544868]
  We use singular vector canonical correlation analysis to study what kind of information is induced from each source. We observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
  arXiv  Detail & Related papers  (2020-04-30T16:25:39Z)
• Evaluating Transformer-Based Multilingual Text Classification [55.53547556060537]
  We argue that NLP tools perform unequally across languages with different syntactic and morphological structures. We calculate word order and morphological similarity indices to aid our empirical study.
  arXiv  Detail & Related papers  (2020-04-29T03:34:53Z)
• Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity [67.36239720463657]
  Multi-SimLex is a large-scale lexical resource and evaluation benchmark covering datasets for 12 diverse languages. Each language dataset is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs. Owing to the alignment of concepts across languages, we provide a suite of 66 cross-lingual semantic similarity datasets.
  arXiv  Detail & Related papers  (2020-03-10T17:17:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.