Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text

• URL: http://arxiv.org/abs/2108.01861v1
• Date: Wed, 4 Aug 2021 06:02:46 GMT
• Title: Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text
• Authors: Vivek Srivastava and Mayank Singh
• Abstract summary: We synthetically generate code-mixed Hinglish sentences using two distinct approaches. We employ human annotators to rate the generation quality.
• Score: 1.6675267471157407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this shared task, we seek the participating teams to investigate the factors influencing the quality of the code-mixed text generation systems. We synthetically generate code-mixed Hinglish sentences using two distinct approaches and employ human annotators to rate the generation quality. We propose two subtasks, quality rating prediction and annotators' disagreement prediction of the synthetic Hinglish dataset. The proposed subtasks will put forward the reasoning and explanation of the factors influencing the quality and human perception of the code-mixed text.

Related papers

• From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences [18.53327811304381]
  Modelling human judgements for the acceptability of code-mixed text can help in distinguishing natural code-mixed text. Cline is the largest of its kind with 16,642 sentences, consisting of samples sourced from two sources. Experiments using Cline demonstrate that simple Multilayer Perceptron (MLP) models trained solely on code-mixing metrics are outperformed by fine-tuned Multilingual Large Language Models (MLLMs)
  arXiv  Detail & Related papers  (2024-05-09T06:40:39Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Language Model Decoding as Direct Metrics Optimization [87.68281625776282]
  Current decoding methods struggle to generate texts that align with human texts across different aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts. We prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts.
  arXiv  Detail & Related papers  (2023-10-02T09:35:27Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation [77.44921096644698]
  We present a systematic analysis of the impact of decoding algorithms on LM fairness. We analyze the trade-off between fairness, diversity and quality.
  arXiv  Detail & Related papers  (2022-10-07T21:33:34Z)
• niksss at HinglishEval: Language-agnostic BERT-based Contextual Embeddings with Catboost for Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text [0.0]
  This paper describes the system description for the HinglishEval challenge at INLG 2022. The goal of this task was to investigate the factors influencing the quality of the code-mixed text generation system.
  arXiv  Detail & Related papers  (2022-06-17T17:36:03Z)
• BITS Pilani at HinglishEval: Quality Evaluation for Code-Mixed Hinglish Text Using Transformers [1.181206257787103]
  This paper aims to determine the factors influencing the quality of Code-Mixed text data generated by the system. For the HinglishEval task, the proposed model uses multi-lingual BERT to find the similarity between synthetically generated and human-generated sentences.
  arXiv  Detail & Related papers  (2022-06-17T10:36:50Z)
• PreCogIIITH at HinglishEval : Leveraging Code-Mixing Metrics & Language Model Embeddings To Estimate Code-Mix Quality [18.806186479627335]
  We attempt to build models that impact the quality of synthetically generated code-mix text by predicting ratings for code-mix quality. In our submission to HinglishEval, a shared-task collocated with INLG2022, we attempt to build models that impact the quality of synthetically generated code-mix text by predicting ratings for code-mix quality.
  arXiv  Detail & Related papers  (2022-06-16T08:00:42Z)
• A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation [79.98319703471596]
  We propose Composition Sampling, a simple but effective method to generate diverse outputs for conditional generation of higher quality. It builds on recently proposed plan-based neural generation models that are trained to first create a composition of the output and then generate by conditioning on it and the input.
  arXiv  Detail & Related papers  (2022-03-28T21:24:03Z)
• Efficient text generation of user-defined topic using generative adversarial networks [0.32228025627337864]
  We propose a User-Defined GAN (UD-GAN) with two-level discriminators to solve this problem. The proposed method is capable of generating texts with less time than others.
  arXiv  Detail & Related papers  (2020-06-22T04:49:47Z)
• Self-Adversarial Learning with Comparative Discrimination for Text Generation [111.18614166615968]
  We propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity.
  arXiv  Detail & Related papers  (2020-01-31T07:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.