Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?

• URL: http://arxiv.org/abs/2409.09568v1
• Date: Sun, 15 Sep 2024 01:06:07 GMT
• Title: Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?
• Authors: Josef Jon,
• Abstract summary: We ask whether the algorithms used in Neural Machine Translation have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. We conduct a series of experiments to investigate whether NMT systems struggle with maintaining the diversity of such texts. Our ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output.
• Score: 3.8073142980733
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and produce. With the wide adoption of these tools in rapid progress, we must ask what these limitations are and what are the possible implications of integrating such tools even more deeply into our daily lives. As a testbed, we have chosen the task of Neural Machine Translation (NMT). Nevertheless, we aim for general insights and outcomes, applicable even to current Large Language Models (LLMs). We ask whether the algorithms used in NMT have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. To explore this hypothesis, we define a set of measures to quantify text diversity based on its statistical properties, like uniformity or rhythmicity of word-level surprisal, on multiple scales (sentence, discourse, language). We then conduct a series of experiments to investigate whether NMT systems struggle with maintaining the diversity of such texts, potentially reducing the richness of the language generated by these systems, compared to human translators. We search for potential causes of these limitations rooted in training objectives and decoding algorithms. Our ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.

Related papers

• On Uncertainty In Natural Language Processing [2.5076643086429993]
  This thesis studies how uncertainty in natural language processing can be characterized from a linguistic, statistical and neural perspective. We propose a method for calibrated sampling in natural language generation based on non-exchangeable conformal prediction. Lastly, we develop an approach to quantify confidence in large black-box language models using auxiliary predictors.
  arXiv  Detail & Related papers  (2024-10-04T14:08:02Z)
• Beyond Turing: A Comparative Analysis of Approaches for Detecting Machine-Generated Text [1.919654267936118]
  Traditional shallow learning, Language Model (LM) fine-tuning, and Multilingual Model fine-tuning are evaluated. Results reveal considerable differences in performance across methods. This study paves the way for future research aimed at creating robust and highly discriminative models.
  arXiv  Detail & Related papers  (2023-11-21T06:23:38Z)
• MAGE: Machine-generated Text Detection in the Wild [82.70561073277801]
  Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection. We build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:13:29Z)
• Why is constrained neural language generation particularly challenging? [13.62873478165553]
  We present an extensive survey on the emerging topic of constrained neural language generation. We distinguish between conditions and constraints, present constrained text generation tasks, and review existing methods and evaluation metrics for constrained text generation. Our aim is to highlight recent progress and trends in this emerging field, informing on the most promising directions and limitations towards advancing the state-of-the-art of constrained neural language generation research.
  arXiv  Detail & Related papers  (2022-06-11T02:07:33Z)
• Exploring Dimensionality Reduction Techniques in Multilingual Transformers [64.78260098263489]
  This paper gives a comprehensive account of the impact of dimensional reduction techniques on the performance of state-of-the-art multilingual Siamese Transformers. It shows that it is possible to achieve an average reduction in the number of dimensions of $91.58% pm 2.59%$ and $54.65% pm 32.20%$, respectively.
  arXiv  Detail & Related papers  (2022-04-18T17:20:55Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)
• When Does Translation Require Context? A Data-driven, Multilingual Exploration [71.43817945875433]
  proper handling of discourse significantly contributes to the quality of machine translation (MT) Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation. We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
  arXiv  Detail & Related papers  (2021-09-15T17:29:30Z)
• SML: a new Semantic Embedding Alignment Transformer for efficient cross-lingual Natural Language Inference [71.57324258813674]
  The ability of Transformers to perform with precision a variety of tasks such as question answering, Natural Language Inference (NLI) or summarising, have enable them to be ranked as one of the best paradigms to address this kind of tasks at present. NLI is one of the best scenarios to test these architectures, due to the knowledge required to understand complex sentences and established a relation between a hypothesis and a premise. In this paper, we propose a new architecture, siamese multilingual transformer, to efficiently align multilingual embeddings for Natural Language Inference.
  arXiv  Detail & Related papers  (2021-03-17T13:23:53Z)
• Intrinsic Probing through Dimension Selection [69.52439198455438]
  Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted.
  arXiv  Detail & Related papers  (2020-10-06T15:21:08Z)
• Informed Sampling for Diversity in Concept-to-Text NLG [8.883733362171034]
  We propose an Imitation Learning approach to explore the level of diversity that a language generation model can reliably produce. Specifically, we augment the decoding process with a meta-classifier trained to distinguish which words at any given timestep will lead to high-quality output.
  arXiv  Detail & Related papers  (2020-04-29T17:43:24Z)
• On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation [2.132096006921048]
  We investigate the discrepancies between the strengths of statistical approaches to machine translation and the way humans translate. We identify linguistic information that is lacking in order for automatic translation systems to produce more accurate translations. We identify overgeneralization or 'algomic bias' as a potential drawback of neural MT and link it to many of the remaining linguistic issues.
  arXiv  Detail & Related papers  (2020-03-31T16:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.