Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese

• URL: http://arxiv.org/abs/2005.00842v1
• Date: Sat, 2 May 2020 14:32:40 GMT
• Title: Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese
• Authors: Tatsuki Kuribayashi, Takumi Ito, Jun Suzuki, Kentaro Inui
• Abstract summary: We examine a methodology using neural language models (LMs) for analyzing the word order of language. We explore whether the LM-based method is valid for analyzing the word order. We conclude that LMs display sufficient word order knowledge for usage as an analysis tool.
• Score: 45.80297329300326
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of preprocessor errors in count-based methods. In this study, we explore whether the LM-based method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool. Finally, using the LM-based method, we demonstrate the relationship between the canonical word order and topicalization, which had yet to be analyzed by large-scale experiments.

Related papers

• How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments [22.76776244036282]
  The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs. The existing dominant approach, however, naively calculates and compares the probabilities of paired sentences using LMs. We investigate how to make the most of LLMs' grammatical knowledge to comprehensively evaluate it.
  arXiv  Detail & Related papers  (2024-08-19T01:53:47Z)
• LLMs for XAI: Future Directions for Explaining Explanations [50.87311607612179]
  We focus on refining explanations computed using existing XAI algorithms. Initial experiments and user study suggest that LLMs offer a promising way to enhance the interpretability and usability of XAI.
  arXiv  Detail & Related papers  (2024-05-09T19:17:47Z)
• Zero-shot Causal Graph Extrapolation from Text via LLMs [50.596179963913045]
  We evaluate the ability of large language models (LLMs) to infer causal relations from natural language. LLMs show competitive performance in a benchmark of pairwise relations without needing (explicit) training samples. We extend our approach to extrapolating causal graphs through iterated pairwise queries.
  arXiv  Detail & Related papers  (2023-12-22T13:14:38Z)
• LM-Polygraph: Uncertainty Estimation for Language Models [71.21409522341482]
  Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of large language models (LLMs) We introduce LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. It introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores.
  arXiv  Detail & Related papers  (2023-11-13T15:08:59Z)
• Evaluating Neural Language Models as Cognitive Models of Language Acquisition [4.779196219827507]
  We argue that some of the most prominent benchmarks for evaluating the syntactic capacities of neural language models may not be sufficiently rigorous. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.
  arXiv  Detail & Related papers  (2023-10-31T00:16:17Z)
• Investigating the Effect of Language Models in Sequence Discriminative Training for Neural Transducers [36.60689278751483]
  We investigate the effect of language models (LMs) with different context lengths and label units (phoneme vs. word) used in sequence discriminative training. Experimental results on Librispeech show that using the word-level LM in training outperforms the phoneme-level LM. Our results reveal the pivotal importance of the hypothesis space quality in sequence discriminative training.
  arXiv  Detail & Related papers  (2023-10-11T09:53:17Z)
• Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models [59.89454513692417]
  Tabular data is often hidden in text, particularly in medical diagnostic reports. We propose a novel, simple, and effective methodology for extracting structured tabular data from textual medical reports, called TEMED-LLM. We demonstrate that our approach significantly outperforms state-of-the-art text classification models in medical diagnostics.
  arXiv  Detail & Related papers  (2023-06-08T09:12:28Z)
• The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generation [23.711953448400514]
  We inspect to which extent neural language models (LMs) exhibit uncertainty over such analyses. We find that LMs can track multiple analyses simultaneously. As a response to disambiguating cues, the LMs often select the correct interpretation, but occasional errors point to potential areas of improvement.
  arXiv  Detail & Related papers  (2021-09-16T10:27:05Z)
• Language Models Use Monotonicity to Assess NPI Licensing [8.856422030608188]
  We investigate the semantic knowledge of language models (LMs) We focus on whether these LMs create categories of linguistic environments based on their semantic monotonicity properties, and whether these categories play a similar role in LMs as in human language understanding.
  arXiv  Detail & Related papers  (2021-05-28T13:32:00Z)
• Language Model Prior for Low-Resource Neural Machine Translation [85.55729693003829]
  We propose a novel approach to incorporate a LM as prior in a neural translation model (TM) We add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior. Results on two low-resource machine translation datasets show clear improvements even with limited monolingual data.
  arXiv  Detail & Related papers  (2020-04-30T16:29:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.