Avaliação de eficiência na leitura: uma abordagem baseada em PLN

• URL: http://arxiv.org/abs/2508.15824v1
• Date: Mon, 18 Aug 2025 02:21:12 GMT
• Title: Avaliação de eficiência na leitura: uma abordagem baseada em PLN
• Abstract summary: This study proposes an automated evaluation model for the cloze test in Brazilian Portuguese.<n>The integrated method demonstrated its effectiveness, achieving a high correlation with human evaluation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The cloze test, widely used due to its low cost and flexibility, makes it possible to assess reading comprehension by filling in gaps in texts, requiring the mobilization of diverse linguistic repertoires. However, traditional correction methods, based only on exact answers, limit the identification of nuances in student performance. This study proposes an automated evaluation model for the cloze test in Brazilian Portuguese, integrating orthographic (edit distance), grammatical (POS tagging) and semantic (similarity between embeddings) analyses. The integrated method demonstrated its effectiveness, achieving a high correlation with human evaluation (0.832). The results indicate that the automated approach is robust, sensitive to variations in linguistic repertoire and suitable for educational contexts that require scalability.

Related papers

• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts [0.0]
  This study aimed to classify Estonian proficiency examination writings (levels A2-C1)<n>Various linguistic properties of the training data were analyzed to identify relevant proficiency predictors.<n>The results have been implemented in the writing evaluation module of an Estonian open-source language learning environment.
  arXiv  Detail & Related papers  (2026-02-13T17:06:17Z)
• Linguistic Indicators of Early Cognitive Decline in the DementiaBank Pitt Corpus: A Statistical and Machine Learning Study [4.417564179511245]
  This study analyzes spontaneous speech transcripts from the DementiaBank Pitt Corpus using three linguistic representations.<n> syntactic and grammatical features retain strong discriminative power even in the absence of lexical content.<n>This study supports the use of linguistically grounded features for transparent and reliable language-based cognitive screening.
  arXiv  Detail & Related papers  (2026-02-11T16:53:57Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Large Language Models for Full-Text Methods Assessment: A Case Study on Mediation Analysis [15.98124151893659]
  Large language models (LLMs) offer potential for automating methodological assessments.<n>We benchmarked state-of-the-art LLMs against expert human reviewers across 180 full-text scientific articles.
  arXiv  Detail & Related papers  (2025-10-12T19:04:22Z)
• Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection [71.59834293521074]
  We develop a framework to distinguish between human-authored and machine-generated text.<n>Our method achieves 98.3% AUROC and AUPR with only 8.9% FPR95 on DeepFake dataset.<n>Code, pretrained weights, and demo will be released.
  arXiv  Detail & Related papers  (2025-10-07T08:14:45Z)
• I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluations [9.275967682881944]
  This paper introduces a comprehensive benchmark for evaluating how Large Language Models respond to linguistic shibboleths.<n>We demonstrate how LLMs systematically penalize certain linguistic patterns, particularly hedging language, despite equivalent content quality.<n>We validate our approach along multiple linguistic dimensions, showing that hedged responses receive 25.6% lower ratings on average.
  arXiv  Detail & Related papers  (2025-08-06T23:51:03Z)
• Enhancing Essay Cohesion Assessment: A Novel Item Response Theory Approach [0.7845950813414773]
  This work proposes and analyses the performance of a cohesion score prediction approach based on item response theory.<n>The proposed approach outperforms conventional machine learning models and ensemble methods in several evaluation metrics.
  arXiv  Detail & Related papers  (2025-07-11T11:05:27Z)
• Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learning [63.531262595858]
  Divide-and-conquer approach breaks comprehensive evaluation task into localized scoring tasks, followed by a final global assessment.<n>We introduce a hybrid in-context learning approach that leverages human annotations to enhance the performance of both local and global evaluations.<n>Finally, we develop an uncertainty-based active learning algorithm that efficiently selects data samples for human annotation.
  arXiv  Detail & Related papers  (2025-05-26T16:39:41Z)
• NLP and Education: using semantic similarity to evaluate filled gaps in a large-scale Cloze test in the classroom [0.0]
  Using data from Cloze tests administered to students in Brazil, WE models for Brazilian Portuguese (PT-BR) were employed to measure semantic similarity. A comparative analysis between the WE models' scores and the judges' evaluations revealed that GloVe was the most effective model.
  arXiv  Detail & Related papers  (2024-11-02T15:22:26Z)
• Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation [41.66053021998106]
  Aspect-based sentiment analysis (ABSA) is a challenging task.<n>Traditional evaluation methods often constrain ground truths (GT) to a single term.<n>We propose a novel and fully automated pipeline that expands existing evaluation sets by adding alternative valid terms for aspect and opinion.
  arXiv  Detail & Related papers  (2024-10-13T11:48:09Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)
• Self-Normalized Importance Sampling for Neural Language Modeling [97.96857871187052]
  In this work, we propose self-normalized importance sampling. Compared to our previous work, the criteria considered in this work are self-normalized and there is no need to further conduct a correction step. We show that our proposed self-normalized importance sampling is competitive in both research-oriented and production-oriented automatic speech recognition tasks.
  arXiv  Detail & Related papers  (2021-11-11T16:57:53Z)
• Language Model Evaluation in Open-ended Text Generation [0.76146285961466]
  We study different evaluation metrics that have been proposed to evaluate quality, diversity and consistency of machine-generated text. From there, we propose a practical pipeline to evaluate language models in open-ended generation task.
  arXiv  Detail & Related papers  (2021-08-08T06:16:02Z)
• TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing [73.16475763422446]
  We propose a multilingual robustness evaluation platform for NLP tasks (TextFlint) It incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analysis. TextFlint generates complete analytical reports as well as targeted augmented data to address the shortcomings of the model's robustness.
  arXiv  Detail & Related papers  (2021-03-21T17:20:38Z)
• Morphologically Aware Word-Level Translation [82.59379608647147]
  We propose a novel morphologically aware probability model for bilingual lexicon induction. Our model exploits the basic linguistic intuition that the lexeme is the key lexical unit of meaning.
  arXiv  Detail & Related papers  (2020-11-15T17:54:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.