When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW

• URL: http://arxiv.org/abs/2607.11213v1
• Date: Mon, 13 Jul 2026 08:07:30 GMT
• Title: When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW
• Abstract summary: High-stakes English proficiency tests treat standardized, unaided performance as evidence for score interpretations about academic English proficiency.<n>This conceptual validity argument reframes AI as a score-interpretation problem in high-stakes language testing.<n>The paper argues that score interpretations should be narrowed and supplemented when used to support claims about AI-mediated academic communication.
• Score: 1.2333280983346018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-stakes English proficiency tests treat standardized, unaided performance as evidence for score interpretations about academic English proficiency. This interpretation remains meaningful, but as target language use domains increasingly involve generative AI, the extrapolation from unaided test performance to academic communicative readiness becomes less self-evident. This conceptual validity argument reframes AI as a score-interpretation problem in high-stakes language testing, not only an operational issue of scoring, feedback, security, or misconduct. Synthesizing current literature in three uneven layers, the paper shows that most work treats AI as assessment infrastructure, while far less theorizes its implications for construct validity and extrapolation warrants. It defines AI-mediated construct drift as the misalignment that arises when communicative abilities required in the target domain change through AI mediation while test constructs remain anchored to an unaided-performance model. It proposes bounded AI mediation as a validity-oriented design principle: a standardized condition in which all test takers access the same institutionally controlled AI assistant, with predefined assistance boundaries, logged interactions, and tasks that distinguish comprehension support from answer generation. The paper argues that score interpretations should be narrowed and supplemented when used to support claims about AI-mediated academic communication.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Toward AI-Resilient Assessment in Computer Science Courses in an AI-Native World [29.886304642534743]
  AI-native course assessments should grade students by their ability to achieve outcomes beyond a strong AI baseline.<n>This paper proposes a minimal formal framework for this goal.<n>We describe an AI-resilient approximate-membership assignment centered on Bloom filters for COMP 480/580 at Rice University.
  arXiv  Detail & Related papers  (2026-06-16T04:54:15Z)
• Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Design [0.0]
  Machine interpreting (MI) is the live, real-time application of speech translation.<n>Yet the user experience remains far inferior to interpreter-mediated communication.<n>This paper defines MI as a distinct subfield of speech translation.
  arXiv  Detail & Related papers  (2026-06-14T20:41:59Z)
• Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition [52.63676763985825]
  We propose a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy.<n>We design an agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs.<n>We will release the code to facilitate future research in interactive and agentic ASR.
  arXiv  Detail & Related papers  (2026-04-10T09:02:42Z)
• When Contextual Inference Fails: Cancelability in Interactive Instruction Following [51.2195840589474]
  We introduce Build What I Mean, an interactive benchmark for contextual meaning construction.<n>In BWIM, models must resolve ambiguity by either performing a contextual inference or requesting clarification at a small communication cost.<n>We observe suboptimal strategies, such as partner-blind over-clarification and question-averse guessing under uncertainty.
  arXiv  Detail & Related papers  (2026-03-20T14:46:59Z)
• From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring [0.0]
  We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI.<n>The evidence needed in the generative AI context is more extensive than in the feature-based scoring context because of the lack of transparency and other unique concerns to generative AI such as consistency.
  arXiv  Detail & Related papers  (2026-03-01T23:21:18Z)
• Beyond Static Scoring: Enhancing Assessment Validity via AI-Generated Interactive Verification [0.4260312058817663]
  Large Language Models (LLMs) challenge the validity of traditional open-ended assessments by blurring the lines of authorship.<n>This paper introduces a novel Human-AI Collaboration framework that enhances assessment integrity by combining rubric-based automated scoring with AI-generated, targeted follow-up questions.
  arXiv  Detail & Related papers  (2025-12-14T08:13:53Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications [0.0]
  generative AI is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring.<n>We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI.
  arXiv  Detail & Related papers  (2025-01-04T16:59:29Z)
• Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents [80.5213198675411]
  Large language models (LLMs) have dramatically enhanced the field of language intelligence. LLMs leverage the intriguing chain-of-thought (CoT) reasoning techniques, obliging them to formulate intermediate steps en route to deriving an answer. Recent research endeavors have extended CoT reasoning methodologies to nurture the development of autonomous language agents.
  arXiv  Detail & Related papers  (2023-11-20T14:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.