PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback

• URL: http://arxiv.org/abs/2606.20287v1
• Date: Thu, 18 Jun 2026 14:29:13 GMT
• Title: PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback
• Abstract summary: PsyScore is a psychometrically-aware framework that integrates diagnostic assessment with instructional scaffolding through a shared latent ability representation.<n> Experiments on the ASAP++ dataset demonstrate that PsyScore achieves competitive scoring performance while providing more pedagogically aligned feedback.
• Score: 13.817126458402017
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective Automated Essay Scoring (AES) are expected to support both reliable assessment and actionable instructional feedback. However, existing approaches often treat scoring and feedback as separate components: neural scoring models provide limited interpretability, while Large Language Model (LLM)-based feedback is typically insensitive to learners proficiency levels. To address this fragmentation, this work proposes PsyScore, a psychometrically-aware framework that integrates diagnostic assessment with instructional scaffolding through a shared latent ability representation. PsyScore comprises three key modules: a Trait-Adaptive Neural IRT Scorer that incorporates the Graded Partial Credit Model (GPCM) into a neural architecture, enabling the precise estimation of student ability while maintaining psychometric interpretability, a ZPD-Scaffolded Feedback Generator, which conditions multi-agent feedback strategies on the diagnosed ability parameter to adapt instructional focus across different proficiency levels, and a Multi-Perspective Feedback Evaluation Strategy that assesses feedback quality via pairwise preference judgements and student revision simulations. Experiments on the ASAP++ dataset demonstrate that PsyScore achieves competitive scoring performance while providing more pedagogically aligned feedback.

Related papers

• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• An Interpretable Closed-Loop Intelligent Tutoring System for Multimodal Affective Feedback in Asynchronous Presentation Training [0.0]
  The ITS maps multimodal inputs into evidence-based feedback that can be traced back to observable performance cues.<n>The system achieved rubric-aligned scoring with performance levels comparable to expert ratings.
  arXiv  Detail & Related papers  (2026-05-17T14:12:40Z)
• Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation [50.85337196571056]
  Text-to-image (T2I) generation has advanced rapidly, making reliable evaluation critical as performance differences between models narrow.<n>Existing evaluation practices typically apply uniform annotation mechanisms, such as Likert-scale or binary question answering (BQA)<n>We revisit T2I evaluation through the lens of skill-aligned annotation, where annotation strategies reflect the underlying characteristics of each evaluation skill.
  arXiv  Detail & Related papers  (2026-05-13T09:14:31Z)
• Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs [1.3819918262591997]
  rank-based correlation metrics obscure intrinsic intercorrelations among analytic dimensions.<n>High correlations may mask a system's true diagnostic behaviour.<n>We propose a self-referential assessment framework that focuses on identifying intra-learner strengths and weaknesses.
  arXiv  Detail & Related papers  (2026-05-05T21:05:25Z)
• Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition [52.63676763985825]
  We propose a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy.<n>We design an agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs.<n>We will release the code to facilitate future research in interactive and agentic ASR.
  arXiv  Detail & Related papers  (2026-04-10T09:02:42Z)
• S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings [2.010691774041283]
  evaluating student responses from long essays to short factual answers is a key challenge in educational NLP.<n>We introduce S-GRADES, a web-based benchmark that consolidates 14 diverse grading datasets under a unified interface.<n>To demonstrate the utility of S-GRADES, we evaluate three state-of-the-art large language models across the benchmark using multiple reasoning strategies in prompting.
  arXiv  Detail & Related papers  (2026-03-10T21:11:51Z)
• A Closed-Loop Personalized Learning Agent Integrating Neural Cognitive Diagnosis, Bounded-Ability Adaptive Testing, and LLM-Driven Feedback [5.190121417265426]
  This paper presents an end-to-end personalized learning agent, which integrates a Neural Cognitive Diagnosis model (NCD), a Bounded-Ability Computerized Adaptive Testing strategy (BECAT), and large language models (LLMs)<n> Experiments on the ASSISTments dataset show that the NCD module achieves strong performance on response prediction while yielding interpretable mastery assessments.<n>Overall, the results indicate that the proposed design is effective and practically deployable.
  arXiv  Detail & Related papers  (2025-10-26T07:32:31Z)
• MOSAIC-F: A Framework for Enhancing Students' Oral Presentation Skills through Personalized Feedback [1.0835264351334324]
  This framework integrates Multimodal Learning Analytics (MMLA), Observations, Sensors, Artificial Intelligence (AI), and Collaborative assessments.<n>By combining human-based and data-based evaluation techniques, this framework enables more accurate, personalized and actionable feedback.
  arXiv  Detail & Related papers  (2025-06-10T09:46:31Z)
• SCAN: Structured Capability Assessment and Navigation for LLMs [54.54085382131134]
  textbfSCAN (Structured Capability Assessment and Navigation) is a practical framework that enables detailed characterization of Large Language Models.<n>SCAN incorporates four key components:.<n>TaxBuilder, which extracts capability-indicating tags from queries to construct a hierarchical taxonomy;.<n>RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag;.<n>A PC$2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach achieves significantly higher accuracy compared to classic LLM-as-a-Judge method
  arXiv  Detail & Related papers  (2025-05-10T16:52:40Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach [84.02388020258141]
  We propose a new framework named ENIGMA for estimating human evaluation scores based on off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.
  arXiv  Detail & Related papers  (2021-02-20T03:29:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.