A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments

• URL: http://arxiv.org/abs/2608.24825v1
• Date: Tue, 25 Aug 2026 17:07:16 GMT
• Title: A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments
• Abstract summary: This study proposes a dual-dimensional framework for Automated Item Similarity Analysis powered by Large Language Models (LLMs)<n>LLMs operationalize similarity through Structured Decomposition and Semantic Relatedness.<n>Psychometric validation indicates that LLM-derived metrics align more closely with indicators of construct-irrelevant local dependence.
• Score: 2.2404539130909376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid expansion of large-scale assessments and the growing adoption of automatic item generation have intensified concerns about incidental content redundancy, where construct-irrelevant elements such as wording or contextual framing become unintentionally repetitive across items. Traditional similarity metrics like BLEU or cosine similarity, often fail to capture the nuanced structural and semantic layers that drive perceived redundancy simultaneously. This study proposes a dual-dimensional framework for Automated Item Similarity Analysis (AISA) powered by Large Language Models (LLMs), operationalizing similarity through Structured Decomposition and Semantic Relatedness. Psychometric validation indicates that LLM-derived metrics align more closely with indicators of construct-irrelevant local dependence and yield more coherent item parameter groupings than traditional text-based measures. The framework is further evaluated through its application in Computerized Adaptive Testing (CAT). Simulations reveal that incorporating LLM-based similarity constraints into item selection improves estimation stability and reduces bias with minimal efficiency trade-offs, outperforming constraints based on conventional metrics. These findings highlight the potential of LLM-powered AISA to support scalable bank curation, content-aware test assembly, and experience-sensitive adaptive testing across diverse assessment contexts.

Related papers

• The Illusion of Multi-Agent Advantage [63.92466143571383]
  We show that expert-architected Multi-Agent Systems (MAS) consistently outperforms automatically generated architectures in raw performance and cost-efficiency.<n>Existing evaluation frameworks mask critical architectural gaps and inefficiencies by failing to account for the marginal utility of increased computational cost.
  arXiv  Detail & Related papers  (2026-06-11T07:39:24Z)
• On the Limitations of Large Language Models for Conceptual Database Modeling [0.0]
  This article analyzes the use of Large Language Models (LLMs) as support for conceptual modeling of relational databases.<n>The approach combines different language models with prompt engineering techniques to evaluate their ability to identify entities.
  arXiv  Detail & Related papers  (2026-05-12T11:35:59Z)
• SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization [55.63565289584336]
  SimReg is an embedding similarity regularization loss that explicitly encourages token representations with the same ground-truth label within each sequence to be more similar.<n>Our analysis reveals that this mechanism introduces gains by enlarging multi-classification margins, thereby enabling more efficient classification.
  arXiv  Detail & Related papers  (2026-05-09T08:59:13Z)
• Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation [2.464003792743989]
  This paper proposes a generalized Multi-Trait Multi-Method (MTMM) framework for Large Language Models (LLMs) evaluation.<n>We formalize and unify nine evaluation metrics, including Paraphrase Instability, Drift Score, Overton Width, and Pluralism Score, interpreting them as geometric measurements within a shared latent coordinate space.<n>By systematically separating task-irrelevant perturbations from true capability spans, the framework provides a theoretically grounded and domain-agnostic taxonomy for robust and empirically stable benchmark design.
  arXiv  Detail & Related papers  (2026-05-08T22:05:19Z)
• SERE: Structural Example Retrieval for Enhancing LLMs in Event Causality Identification [50.174050426976514]
  Event Causality Identification (ECI) requires models to determine whether a given pair of events in a context exhibits a causal relationship.<n>Large Language Models (LLMs) have demonstrated strong performance across various NLP tasks, but their effectiveness in ECI remains limited due to biases in causal reasoning.<n>We propose SERE, a structural example retrieval framework that leverages LLMs' few-shot learning capabilities.
  arXiv  Detail & Related papers  (2026-05-05T12:50:19Z)
• Transforming External Knowledge into Triplets for Enhanced Retrieval in RAG of LLMs [55.78708003681562]
  Retrieval-Augmented Generation (RAG) mitigates hallucination in large language models (LLMs) by incorporating external knowledge during generation.<n>Existing RAG approaches typically retrieve and unstructured text fragments as context.<n>We propose Tri-RAG, a structured triplet-based retrieval framework that improves retrieval efficiency through reasoning-aligned context construction.
  arXiv  Detail & Related papers  (2026-04-14T11:36:29Z)
• RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents [4.07447364754644]
  This paper introduces RPA-Check, a multi-stage automated evaluation framework designed to objectively assess the performance of LLM-based RPAs.<n>We validate this framework by applying it to LLM Court, a serious game for forensic training involving several quantized local models.
  arXiv  Detail & Related papers  (2026-04-13T16:08:03Z)
• Table-BiEval: A Self-Supervised, Dual-Track Framework for Decoupling Structure and Content in LLM Evaluation [11.450834626205676]
  Table-BiEval is a novel approach based on a human-free, self-supervised evaluation framework.<n>It calculates Content Semantic Accuracy and Normalized Tree Edit Distance to decouple structure from content.<n>Results reveal substantial variability, highlighting that mid-sized models can surprisingly outperform larger counterparts in structural efficiency.
  arXiv  Detail & Related papers  (2026-01-09T07:38:27Z)
• Accelerate Speculative Decoding with Sparse Computation in Verification [49.74839681322316]
  Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel.<n>Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding.<n>We propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost.
  arXiv  Detail & Related papers  (2025-12-26T07:53:41Z)
• NDCG-Consistent Softmax Approximation with Accelerated Convergence [67.10365329542365]
  We propose novel loss formulations that align directly with ranking metrics.<n>We integrate the proposed RG losses with the highly efficient Alternating Least Squares (ALS) optimization method.<n> Empirical evaluations on real-world datasets demonstrate that our approach achieves comparable or superior ranking performance.
  arXiv  Detail & Related papers  (2025-06-11T06:59:17Z)
• DCR-Consistency: Divide-Conquer-Reasoning for Consistency Evaluation and Improvement of Large Language Models [4.953092503184905]
  This work proposes DCR, an automated framework for evaluating and improving the consistency of Large Language Models (LLMs) generated texts. We introduce an automatic metric converter (AMC) that translates the output from DCE into an interpretable numeric score. Our approach also substantially reduces nearly 90% of output inconsistencies, showing promise for effective hallucination mitigation.
  arXiv  Detail & Related papers  (2024-01-04T08:34:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.