Assessing the validity of new paradigmatic complexity measures as criterial features for proficiency in L2 writings in English

• URL: http://arxiv.org/abs/2503.10220v2
• Date: Fri, 14 Mar 2025 08:44:13 GMT
• Title: Assessing the validity of new paradigmatic complexity measures as criterial features for proficiency in L2 writings in English
• Abstract summary: This article addresses Second Language (L2) writing development through an investigation of new grammatical and structural complexity metrics.<n>We explore the paradigmatic production in learner English by linking language functions to specific grammatical paradigms.
• Score: 1.465540676497032
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This article addresses Second Language (L2) writing development through an investigation of new grammatical and structural complexity metrics. We explore the paradigmatic production in learner English by linking language functions to specific grammatical paradigms. Using the EFCAMDAT as a gold standard and a corpus of French learners as an external test set, we employ a supervised learning framework to operationalise and evaluate seven microsystems. We show that learner levels are associated with the seven microsystems (MS). Using ordinal regression modelling for evaluation, the results show that all MS are significant but yield a low impact if taken individually. However, their influence is shown to be impactful if taken as a group. These microsystems and their measurement method suggest that it is possible to use them as part of broader-purpose CALL systems focused on proficiency assessment.

Related papers

• A didactical-driven teacher assistant for a dimensional modeling course [1.338174941551702]
  This paper presents a didactical-driven teacher assistant for a French-language university course on dimensional modelling.<n>The architecture formalises the instructor's pedagogical reasoning into deterministic modules that handle intent detection, concept linking, and didactic approach selection.
  arXiv  Detail & Related papers  (2026-06-12T11:35:33Z)
• UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy [99.95663439922937]
  In-context Learning efficacy is often non-monotonic and highly task-dependent.<n>We introduce a six-level capability-oriented taxonomy that categorizes the functional role of demonstrations.<n>We construct UniICL-760K, a large-scale corpus featuring curated 8-shot In-context Learning episodes.<n>As an architectural intervention, we propose the Context-Adaptive Prototype Modulator.
  arXiv  Detail & Related papers  (2026-03-25T18:09:33Z)
• Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs [3.027026792589421]
  The English Grammar Profile (EGP) is a taxonomy of grammatical constructs mapped to the proficiency levels of the Common European Framework of Reference (CEFR)<n>We propose a novel framework leveraging the EGP to detect learners' attempts at grammatical constructs and classify them as successful or unsuccessful.
  arXiv  Detail & Related papers  (2026-03-17T22:06:00Z)
• What Works for 'Lost-in-the-Middle' in LLMs? A Study on GM-Extract and Mitigations [1.2879523047871226]
  GM-Extract is a novel benchmark dataset meticulously designed to evaluate LLM performance on retrieval of control variables.<n>We conduct a systematic evaluation of 7-8B parameter models on two multi-document tasks (key-value extraction and question-answering)<n>While a distinct U-shaped curve was not consistently observed, our analysis reveals a clear pattern of performance across models.
  arXiv  Detail & Related papers  (2025-11-17T20:50:50Z)
• An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Education [0.30723404270319693]
  This study presents two open-source embedding models fine-tuned for educational question answering.<n>A synthetic dataset of 3,197 sentence pairs was constructed through a combination of manual curation and large language model (LLM)-assisted generation.<n>Two training strategies were evaluated: (1) a baseline model fine-tuned using MultipleNegativesRankingLoss (MNRL), and (2) a dual-loss model that combines MNRL with CosineSimilarityLoss to improve both semantic ranking and similarity calibration.
  arXiv  Detail & Related papers  (2025-05-08T03:14:14Z)
• Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey [64.08485471150486]
  This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. We systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication.
  arXiv  Detail & Related papers  (2025-03-28T14:08:40Z)
• Analysis of LLM as a grammatical feature tagger for African American English [0.6927055673104935]
  African American English (AAE) presents unique challenges in natural language processing (NLP)<n>This research systematically compares the performance of available NLP models.<n>This study highlights the necessity for improved model training and architectural adjustments to better accommodate AAE's unique linguistic characteristics.
  arXiv  Detail & Related papers  (2025-02-09T19:46:33Z)
• Advancing Student Writing Through Automated Syntax Feedback [10.137657521054356]
  This study underscores the pivotal role of syntax feedback in augmenting the syntactic proficiency of students.<n>We introduce a specialized dataset named Essay-Syntax-Instruct designed to enhance the understanding and application of English syntax.
  arXiv  Detail & Related papers  (2025-01-13T23:10:02Z)
• MILE: A Mutation Testing Framework of In-Context Learning Systems [5.419884861365132]
  We propose a mutation testing framework designed to characterize the quality and effectiveness of test data for ICL systems. First, we propose several mutation operators specialized for ICL demonstrations, as well as corresponding mutation scores for ICL test sets. With comprehensive experiments, we showcase the effectiveness of our framework in evaluating the reliability and quality of ICL test suites.
  arXiv  Detail & Related papers  (2024-09-07T13:51:42Z)
• Evaluating Large Language Models Using Contrast Sets: An Experimental Approach [0.0]
  We introduce an innovative technique for generating a contrast set for the Stanford Natural Language Inference dataset. Our strategy involves the automated substitution of verbs, adverbs, and adjectives with their synonyms to preserve the original meaning of sentences. This method aims to assess whether a model's performance is based on genuine language comprehension or simply on pattern recognition.
  arXiv  Detail & Related papers  (2024-04-02T02:03:28Z)
• Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models [0.0]
  Leveraging Large Language Models (LLMs) has shown remarkable promise in enhancing summarization techniques. This paper embarks on an exploration of text summarization with a diverse set of LLMs, including MPT-7b-instruct, falcon-7b-instruct, and OpenAI ChatGPT text-davinci-003 models.
  arXiv  Detail & Related papers  (2023-10-16T14:33:02Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Coverage-based Example Selection for In-Context Learning [27.215972147196805]
  We show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects of the test input. On 15 datasets spanning 6 tasks and with 7 diverse LLMs, we show that (1) BSR is the superior metric for in-context example selection across the board, and (2) for compositional tasks, Set-BSR outperforms independent ranking by up to 17 points on average.
  arXiv  Detail & Related papers  (2023-05-24T08:58:28Z)
• Document-Level Machine Translation with Large Language Models [91.03359121149595]
  Large language models (LLMs) can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. This paper provides an in-depth evaluation of LLMs' ability on discourse modeling.
  arXiv  Detail & Related papers  (2023-04-05T03:49:06Z)
• Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning [104.58874584354787]
  In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models.
  arXiv  Detail & Related papers  (2023-01-27T18:59:01Z)
• Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale [52.663117551150954]
  A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
  arXiv  Detail & Related papers  (2020-10-26T13:57:20Z)
• ECML: An Ensemble Cascade Metric Learning Mechanism towards Face Verification [50.137924223702264]
  In particular, hierarchical metric learning is executed in the cascade way to alleviate underfitting. Considering the feature distribution characteristics of faces, a robust Mahalanobis metric learning method (RMML) with closed-form solution is additionally proposed. EC-RMML is superior to state-of-the-art metric learning methods for face verification.
  arXiv  Detail & Related papers  (2020-07-11T08:47:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.