ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical Normalization

• URL: http://arxiv.org/abs/2501.07020v1
• Date: Mon, 13 Jan 2025 02:47:13 GMT
• Title: ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical Normalization
• Abstract summary: ViSoLex is an open-source system designed to address the challenges of lexical normalization for Vietnamese social media text.<n>By publishing the source code, ViSoLex aims to contribute to the development of more robust Vietnamese natural language processing tools.
• Score: 1.053698976085779
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: ViSoLex is an open-source system designed to address the unique challenges of lexical normalization for Vietnamese social media text. The platform provides two core services: Non-Standard Word (NSW) Lookup and Lexical Normalization, enabling users to retrieve standard forms of informal language and standardize text containing NSWs. ViSoLex's architecture integrates pre-trained language models and weakly supervised learning techniques to ensure accurate and efficient normalization, overcoming the scarcity of labeled data in Vietnamese. This paper details the system's design, functionality, and its applications for researchers and non-technical users. Additionally, ViSoLex offers a flexible, customizable framework that can be adapted to various datasets and research requirements. By publishing the source code, ViSoLex aims to contribute to the development of more robust Vietnamese natural language processing tools and encourage further research in lexical normalization. Future directions include expanding the system's capabilities for additional languages and improving the handling of more complex non-standard linguistic patterns.

Related papers

• L-ReLF: A Framework for Lexical Dataset Creation [0.0]
  The lack of standardized terminology poses a critical barrier to knowledge equity in platforms like Wikipedia.<n>The L-ReLF methodology is designed for generalizability, offering other language communities a clear path to build lexical data for downstream NLP applications.
  arXiv  Detail & Related papers  (2026-03-31T07:19:00Z)
• FreeTxt-Vi: A Benchmarked Vietnamese-English Toolkit for Segmentation, Sentiment, and Summarisation [3.0712751539922394]
  FreeTxt-Vi is a free and open source web based toolkit for creating and analysing bilingual Vietnamese English text collections.<n>System combines corpus analysis features such as concordancing keyword analysis word relation exploration and interactive visualisation with transformer based NLP components for sentiment analysis and summarisation.
  arXiv  Detail & Related papers  (2026-03-05T21:26:08Z)
• Privacy-Preserving Real-Time Vietnamese-English Translation on iOS using Edge AI [0.6526824510982802]
  This project confronts challenges in deploying complex NMT models on resource-limited mobile devices.<n>The application is realized as a real-time iOS prototype, tightly integrating modern iOS frameworks and privacy-by-design principles.
  arXiv  Detail & Related papers  (2025-05-12T14:05:39Z)
• Edit Once, Update Everywhere: A Simple Framework for Cross-Lingual Knowledge Synchronization in LLMs [60.12222055772508]
  We present a simple and practical state-of-the-art (SOTA) recipe Cross-Lingual Knowledge Democracy Edit (X-KDE) X-KDE is designed to propagate knowledge from a dominant language to other languages effectively. Experiments on the Bi-ZsRE and MzsRE benchmarks show that X-KDE significantly enhances cross-lingual performance.
  arXiv  Detail & Related papers  (2025-02-20T15:32:31Z)
• Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages [0.0]
  In multilingual societies like India, text often exhibits code-mixing, blending local languages with English at different linguistic levels. This paper introduces a prompt based method for a shared task aimed at addressing word-level LI challenges in Dravidian languages. In this work, we leveraged GPT-3.5 Turbo to understand whether the large language models is able to correctly classify words into correct categories.
  arXiv  Detail & Related papers  (2024-11-06T16:20:37Z)
• A Library for Automatic Natural Language Generation of Spanish Texts [6.102700502396687]
  We present a novel system for natural language generation (NLG) of Spanish sentences from a minimum set of meaningful words. The system is able to generate complete, coherent and correctly spelled sentences from the main word sets presented by the user. It can be easily adapted to other languages by design and can feiblyas be integrated in a wide range of digital devices.
  arXiv  Detail & Related papers  (2024-05-27T15:44:06Z)
• CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
  We propose CodeGRAG, a Graphical Retrieval Augmented Code Generation framework to enhance the performance of LLMs. CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to fill the gap between programming languages and natural language. Various experiments and ablations are done on four datasets including both the C++ and python languages to validate the hard meta-graph prompt, the soft prompting technique, and the effectiveness of the objectives for pretrained GNN expert.
  arXiv  Detail & Related papers  (2024-05-03T02:48:55Z)
• We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Text [8.956635443376527]
  We present a suite of experiments that allow us to understand the underlying challenges of language model adaptation to nonstandard text. We do so by designing interventions that approximate core features of user-generated text and their interactions with existing biases of language models. Applying our interventions during language model adaptation to nonstandard text variations, we gain important insights into when such adaptation is successful.
  arXiv  Detail & Related papers  (2024-04-10T18:56:53Z)
• Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text [1.9185059111021852]
  We investigate how pre-trained Language Models handle code-switched text in three dimensions. Our findings reveal that pre-trained language models are effective in generalising to code-switched text.
  arXiv  Detail & Related papers  (2024-03-07T19:46:03Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• BiSync: A Bilingual Editor for Synchronized Monolingual Texts [2.0411082897313984]
  We present BiSync, a bilingual writing assistant that allows users to freely compose text in two languages. We detail the model architecture used for synchronization and evaluate the resulting tool, showing that high accuracy can be attained with limited computational resources.
  arXiv  Detail & Related papers  (2023-06-01T07:03:47Z)
• Offline RL for Natural Language Generation with Implicit Language Q Learning [87.76695816348027]
  Large language models can be inconsistent when it comes to completing user specified tasks. We propose a novel RL method, that combines both the flexible utility framework of RL with the ability of supervised learning. In addition to empirically validating ILQL, we present a detailed empirical analysis situations where offline RL can be useful in natural language generation settings.
  arXiv  Detail & Related papers  (2022-06-05T18:38:42Z)
• Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation [133.7313847857935]
  Our study highlights how NLP methods can be adapted to thousands more languages that are under-served by current technology. For 19 under-represented languages across 3 tasks, our methods lead to consistent improvements of up to 5 and 15 points with and without extra monolingual text respectively.
  arXiv  Detail & Related papers  (2022-03-17T16:48:22Z)
• A Multi-Perspective Architecture for Semantic Code Search [58.73778219645548]
  We propose a novel multi-perspective cross-lingual neural framework for code--text matching. Our experiments on the CoNaLa dataset show that our proposed model yields better performance than previous approaches.
  arXiv  Detail & Related papers  (2020-05-06T04:46:11Z)
• XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning [68.57658225995966]
  Cross-lingual Choice of Plausible Alternatives (XCOPA) is a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods falls short compared to translation-based transfer.
  arXiv  Detail & Related papers  (2020-05-01T12:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.