Sorting the Babble in Babel: Assessing the Performance of Language Detection Algorithms on the OpenAlex Database

• URL: http://arxiv.org/abs/2502.03627v2
• Date: Tue, 18 Feb 2025 12:19:04 GMT
• Title: Sorting the Babble in Babel: Assessing the Performance of Language Detection Algorithms on the OpenAlex Database
• Abstract summary: This project aims to compare various language classification procedures, procedures combining various Python language detection algorithms and metadata-based corpora extracted from manually-annotated articles sampled from the OpenAlex database.<n>Following an analysis of precision and recall performance for each algorithm, corpus, and language as well as of processing speeds recorded for each algorithm and corpus type, overall procedure performance at the database level was simulated.<n>Results show that procedure performance strongly depends on the importance given to each of the measures implemented.<n>It is hoped that these results help confirm and foster the unparalleled potential of the OpenAlex database for cross-linguistic, bibliometric-
• Score: 1.0124625066746598
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This project aims to compare various language classification procedures, procedures combining various Python language detection algorithms and metadata-based corpora extracted from manually-annotated articles sampled from the OpenAlex database. Following an analysis of precision and recall performance for each algorithm, corpus, and language as well as of processing speeds recorded for each algorithm and corpus type, overall procedure performance at the database level was simulated using probabilistic confusion matrices for each algorithm, corpus, and language as well as a probabilistic model of relative article language frequencies for the whole OpenAlex database. Results show that procedure performance strongly depends on the importance given to each of the measures implemented: for contexts where precision is preferred, using the LangID algorithm on the greedy corpus gives the best results; however, for all cases where recall is considered at least slightly more important than precision or as soon as processing times are given any kind of consideration, the procedure combining the FastSpell algorithm and the Titles corpus outperforms all other alternatives. Given the lack of truly multilingual, large-scale bibliographic databases, it is hoped that these results help confirm and foster the unparalleled potential of the OpenAlex database for cross-linguistic, bibliometric-based research and analysis.

Related papers

• Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature [0.0]
  In this study, we apply a selection of specialised supervised XMLC methods to the test case of subject indexing contemporary German scientific literature.<n>A challenge for all methods is to reliably make suggestions from the long tail of the subject vocabulary.<n>We find that supervised XMLC algorithms relying on transformer-based dense features give best results in terms of overall binary relevance metrics.
  arXiv  Detail & Related papers  (2026-07-16T11:56:20Z)
• Skeletons Matter: Dynamic Data Augmentation for Text-to-Query [66.52311036179294]
  We formally define the Text-to-Query task paradigm, unifying semantic parsing tasks across various query languages.<n>We identify query skeletons as a shared optimization target of Text-to-Query tasks, and propose a general dynamic data augmentation framework.<n> Experiments on four Text-to-Query benchmarks demonstrate that our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-11-24T09:39:03Z)
• Morpheme Induction for Emergent Language [2.1700203922407497]
  CSAR is an algorithm for inducing morphemes from emergent language corpora of parallel utterances and meanings.<n>It is validated on procedurally generated datasets and compared against baselines for related tasks.<n>We analyze a handful of emergent languages, quantifying linguistic characteristics like degree of synonymy and polysemy.
  arXiv  Detail & Related papers  (2025-10-03T18:59:53Z)
• Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled Data [59.30098850050971]
  Cross-lingual transfer learning can improve performance on tasks with limited labeled data.<n>We leverage nearest-neighbor retrieval to augment minimal labeled data in the target language.<n>We evaluate our approach on eight languages and demonstrate that it consistently outperforms models trained solely on the target language data.
  arXiv  Detail & Related papers  (2025-05-20T12:25:33Z)
• SoftMatcha: A Soft and Fast Pattern Matcher for Billion-Scale Corpus Searches [5.80278230280824]
  We propose a novel algorithm that achieves semantic yet efficient pattern matching by relaxing a surface-level matching with word embeddings. Our experiments demonstrate that the proposed method can execute searches on billion-scale corpora in less than a second.
  arXiv  Detail & Related papers  (2025-03-05T17:53:11Z)
• Leveraging Foundation Language Models (FLMs) for Automated Cohort Extraction from Large EHR Databases [50.552056536968166]
  We propose and evaluate an algorithm for automating column matching on two large, popular and publicly-accessible EHR databases.<n>Our approach achieves a high top-three accuracy of $92%$, correctly matching $12$ out of the $13$ columns of interest, when using a small, pre-trained general purpose language model.
  arXiv  Detail & Related papers  (2024-12-16T06:19:35Z)
• From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models [63.188607839223046]
  This survey focuses on the benefits of scaling compute during inference. We explore three areas under a unified mathematical formalism: token-level generation algorithms, meta-generation algorithms, and efficient generation.
  arXiv  Detail & Related papers  (2024-06-24T17:45:59Z)
• Validating and Exploring Large Geographic Corpora [0.76146285961466]
  Three methods are used to improve the quality of sub-corpora representing specific language-country pairs like New Zealand English. The evaluation shows that the validity of sub-corpora is improved with each stage of cleaning but that this improvement is unevenly distributed across languages and populations.
  arXiv  Detail & Related papers  (2024-03-13T02:46:17Z)
• A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets [1.1647644386277962]
  Typologically diverse benchmarks are increasingly created to track the progress achieved in multilingual NLP. We propose assessing linguistic diversity of a data set against a reference language sample.
  arXiv  Detail & Related papers  (2024-03-06T18:14:22Z)
• Dense X Retrieval: What Retrieval Granularity Should We Use? [56.90827473115201]
  Often-overlooked design choice is the retrieval unit in which the corpus is indexed, e.g. document, passage, or sentence. We introduce a novel retrieval unit, proposition, for dense retrieval. Experiments reveal that indexing a corpus by fine-grained units such as propositions significantly outperforms passage-level units in retrieval tasks.
  arXiv  Detail & Related papers  (2023-12-11T18:57:35Z)
• NSOAMT -- New Search Only Approach to Machine Translation [0.0]
  A "new search only approach to machine translation" was adopted to tackle some of the slowness and inaccuracy of the other technologies. The idea is to develop a solution that, by indexing an incremental set of words that combine a certain semantic meaning, makes it possible to create a process of correspondence between their native language record and the language of translation.
  arXiv  Detail & Related papers  (2023-09-19T11:12:21Z)
• T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification [50.675552118811]
  Cross-lingual text classification is typically built on large-scale, multilingual language models (LMs) pretrained on a variety of languages of interest. We propose revisiting the classic "translate-and-test" pipeline to neatly separate the translation and classification stages.
  arXiv  Detail & Related papers  (2023-06-08T07:33:22Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Zero-Shot Listwise Document Reranking with a Large Language Model [58.64141622176841]
  We propose Listwise Reranker with a Large Language Model (LRL), which achieves strong reranking effectiveness without using any task-specific training data. Experiments on three TREC web search datasets demonstrate that LRL not only outperforms zero-shot pointwise methods when reranking first-stage retrieval results, but can also act as a final-stage reranker.
  arXiv  Detail & Related papers  (2023-05-03T14:45:34Z)
• A Comparison of Document Similarity Algorithms [0.0]
  It addresses the most effective document similarity algorithm by categorizing them into 3 types of document similarity algorithms. The most effective algorithms in each category are also compared in our work using a series of benchmark datasets and evaluations.
  arXiv  Detail & Related papers  (2023-04-03T19:50:55Z)
• Scaling Expert Language Models with Unsupervised Domain Discovery [107.08940500543447]
  We introduce a simple but effective method to asynchronously train large, sparse language models on arbitrary text corpora. Our method clusters a corpus into sets of related documents, trains a separate expert language model on each cluster, and combines them in a sparse ensemble for inference.
  arXiv  Detail & Related papers  (2023-03-24T17:38:58Z)
• Evaluating Various Tokenizers for Arabic Text Classification [4.110108749051656]
  We introduce three new tokenization algorithms for Arabic and compare them to three other baselines using unsupervised evaluations. Our experiments show that the performance of such tokenization algorithms depends on the size of the dataset, type of the task, and the amount of morphology that exists in the dataset.
  arXiv  Detail & Related papers  (2021-06-14T16:05:58Z)
• TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing [73.16475763422446]
  We propose a multilingual robustness evaluation platform for NLP tasks (TextFlint) It incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analysis. TextFlint generates complete analytical reports as well as targeted augmented data to address the shortcomings of the model's robustness.
  arXiv  Detail & Related papers  (2021-03-21T17:20:38Z)
• Deep Graph Matching and Searching for Semantic Code Retrieval [76.51445515611469]
  We propose an end-to-end deep graph matching and searching model based on graph neural networks. We first represent both natural language query texts and programming language code snippets with the unified graph-structured data. In particular, DGMS not only captures more structural information for individual query texts or code snippets but also learns the fine-grained similarity between them.
  arXiv  Detail & Related papers  (2020-10-24T14:16:50Z)
• A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation [71.31905141672529]
  We study the widely adopted ancestral sampling algorithms for auto-regressive language models. We identify three key properties that are shared among them: entropy reduction, order preservation, and slope preservation. We find that the set of sampling algorithms that satisfies these properties performs on par with the existing sampling algorithms.
  arXiv  Detail & Related papers  (2020-09-15T17:28:42Z)
• New Oracle-Efficient Algorithms for Private Synthetic Data Release [52.33506193761153]
  We present three new algorithms for constructing differentially private synthetic data. The algorithms satisfy differential privacy even in the worst case. Compared to the state-of-the-art method High-Dimensional Matrix Mechanism citeMcKennaMHM18, our algorithms provide better accuracy in the large workload.
  arXiv  Detail & Related papers  (2020-07-10T15:46:05Z)
• KLEJ: Comprehensive Benchmark for Polish Language Understanding [4.702729080310267]
  We introduce a comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard. We also release HerBERT, a Transformer-based model trained specifically for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks.
  arXiv  Detail & Related papers  (2020-05-01T21:55:40Z)
• An efficient automated data analytics approach to large scale computational comparative linguistics [0.0]
  This research project aimed to overcome the challenge of analysing human language relationships. It developed automated comparison techniques based on the phonetic representation of certain key words and concept. It led to the development of a workflow which was later implemented by combining Unix shell scripts, a developed R package and SWI Prolog.
  arXiv  Detail & Related papers  (2020-01-31T15:25:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.