Validity Verification of the New TOEFL Writing Task Based on Classical Test Theory

• URL: http://arxiv.org/abs/2509.05347v1
• Date: Tue, 02 Sep 2025 14:23:46 GMT
• Title: Validity Verification of the New TOEFL Writing Task Based on Classical Test Theory
• Abstract summary: iBT has introduced the Academic Discussion Task (ADT) to assess test-takers' ability to engage in academic discourse.<n>Research on the ADT's validity and fairness particularly for culturally and linguistically diverse groups, such as Chinese students, remains limited.<n>This study finds a robust correlation between the ADT and the CET-6 writing and translation subscores.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The TOEFL iBT has introduced the Academic Discussion Task (ADT) to assess test-takers' ability to engage in academic discourse, reflecting the growing emphasis on interactive communication skills in higher education. However, research on the ADT's validity and fairness particularly for culturally and linguistically diverse groups, such as Chinese students, remains limited. This study addresses this gap by employing Classical Test Theory (CTT) to evaluate the psychometric properties of the ADT among Chinese university students. This study finds a robust correlation between the ADT and the CET-6 writing and translation subscores. In addition, there is a high level of expert agreement regarding the construct validity evidence and the appropriateness of the scoring rubric. Furthermore, the results indicate that gender differences in validity indices are minimal. Taken together, these results suggest that the ADT is a valid measure for Chinese test-takers without gender discrimination. However, it is recommended that the cultural sensitivity of the scoring rubric be further refined and that the CET-6 subscores for writing be retained for predictive purposes, in order to better accommodate the needs of diverse test-taker populations. By addressing these issues, this study contributes to the broader discourse on fairness and validity in high-stakes language assessments.

Related papers

• What Really Counts? Examining Step and Token Level Attribution in Multilingual CoT Reasoning [0.03499870393443267]
  This study investigates the attribution patterns underlying Chain-of-Thought (CoT) reasoning in multilingual LLMs.<n>We apply two complementary attribution methods--ContextCite for step-level attribution and Inseq for token-level attribution--to the Qwen2.5 1.5B-Instruct model.<n>Our experimental results highlight key findings such as: (1) attribution scores excessively emphasize the final reasoning step, particularly in incorrect generations; (2) structured CoT prompting significantly improves accuracy for high-resource Latin-script languages; and (3) controlled perturbations via negation and distractor sentences reduce model accuracy and attribution coherence.
  arXiv  Detail & Related papers  (2025-11-19T21:23:58Z)
• A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages [48.68444770923683]
  We present the first comprehensive study of multilingual Chain-of-Thought (CoT) reasoning.<n>We measure language compliance, answer accuracy, and answer consistency when LRMs are prompt-hacked to think in a target language.<n>We find that the quality and effectiveness of thinking traces vary substantially depending on the prompt language.
  arXiv  Detail & Related papers  (2025-10-10T17:06:50Z)
• MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation [91.22008265721952]
  MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned benchmark covering 8 Asian countries and 10 languages.<n>This is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech.<n>We propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity.
  arXiv  Detail & Related papers  (2025-10-07T14:12:12Z)
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective [52.452449102961225]
  This study proposes a novel cross-linguistic perspective to investigate reasoning generalization.<n>Our findings reveal that cross-lingual transferability varies significantly across initial model, target language, and training paradigm.<n>Our study challenges the assumption that LRM reasoning mirrors human cognition, providing critical insights for the development of more language-agnostic LRMs.
  arXiv  Detail & Related papers  (2025-10-02T17:49:49Z)
• Exploring AI-Enabled Test Practice, Affect, and Test Outcomes in Language Assessment [0.0]
  Generative AI-driven, automated item generation (AIG) scales the creation of large item banks and multiple practice tests.<n>This study is the first large-scale study exploring the use of AIG-enabled practice tests in high-stakes language assessment.
  arXiv  Detail & Related papers  (2025-08-23T18:41:30Z)
• S-DAT: A Multilingual, GenAI-Driven Framework for Automated Divergent Thinking Assessment [23.509294903995745]
  This paper introduces S- DAT (Synthetic-Divergent Association Task), a scalable, multilingual framework for automated assessment of divergent thinking (DT)<n>We evaluate S- DAT across eleven diverse languages, including English, Spanish, German, Russian, Hindi, and Japanese (Kanji, Hiragana, Katakana)<n>Unlike prior DAT approaches, the S- DAT shows convergent validity with other DT measures and correct discriminant validity with convergent thinking.
  arXiv  Detail & Related papers  (2025-05-14T02:08:40Z)
• NLP and Education: using semantic similarity to evaluate filled gaps in a large-scale Cloze test in the classroom [0.0]
  Using data from Cloze tests administered to students in Brazil, WE models for Brazilian Portuguese (PT-BR) were employed to measure semantic similarity. A comparative analysis between the WE models' scores and the judges' evaluations revealed that GloVe was the most effective model.
  arXiv  Detail & Related papers  (2024-11-02T15:22:26Z)
• STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond [68.47402386668846]
  We introduce Structured Reasoning In Critical Text Assessment (STRICTA) to model text assessment as an explicit, step-wise reasoning process.<n>STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory.<n>We apply STRICTA to a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers.
  arXiv  Detail & Related papers  (2024-09-09T06:55:37Z)
• Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model [50.339632513018934]
  supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences. We critically examine this hypothesis within the scope of cross-lingual generation tasks. We introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens.
  arXiv  Detail & Related papers  (2024-04-25T17:19:36Z)
• LLaMA Beyond English: An Empirical Study on Language Capability Transfer [49.298360366468934]
  We focus on how to effectively transfer the capabilities of language generation and following instructions to a non-English language. We analyze the impact of key factors such as vocabulary extension, further pretraining, and instruction tuning on transfer. We employ four widely used standardized testing benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench.
  arXiv  Detail & Related papers  (2024-01-02T06:29:02Z)
• Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents [80.5213198675411]
  Large language models (LLMs) have dramatically enhanced the field of language intelligence. LLMs leverage the intriguing chain-of-thought (CoT) reasoning techniques, obliging them to formulate intermediate steps en route to deriving an answer. Recent research endeavors have extended CoT reasoning methodologies to nurture the development of autonomous language agents.
  arXiv  Detail & Related papers  (2023-11-20T14:30:55Z)
• The competent Computational Thinking test (cCTt): a valid, reliable and gender-fair test for longitudinal CT studies in grades 3-6 [0.06282171844772422]
  This study investigated whether the competent Computational Thinking test (cCTt) could evaluate learning reliably from grades 3 to 6 (ages 7-11) using data from 2709 students. The findings indicate that the cCTt is valid, reliable and gender-fair for grades 3-6, although more complex items would be beneficial for grades 5-6.
  arXiv  Detail & Related papers  (2023-05-31T03:29:04Z)
• Evaluating the Performance of Large Language Models on GAOKAO Benchmark [53.663757126289795]
  This paper introduces GAOKAO-Bench, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples. With human evaluation, we obtain the converted total score of LLMs, including GPT-4, ChatGPT and ERNIE-Bot. We also use LLMs to grade the subjective questions, and find that model scores achieve a moderate level of consistency with human scores.
  arXiv  Detail & Related papers  (2023-05-21T14:39:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.