AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models

• URL: http://arxiv.org/abs/2606.21832v1
• Date: Sat, 20 Jun 2026 01:55:28 GMT
• Title: AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models
• Abstract summary: We propose AgentCAT, a Large Language Model-based multi-agent simulation system for dynamic testing.<n>This framework comprises three modules: (1) The examinee agent with memory retrieval and Chain-of-Thought reasoning simulates responses based on cognitive profiles; (2) The selection agent uses coarse-to-fine bucketing and knowledge graph exploration to balance local difficulty and global coverage; and (3) The supervisor uses dual-auditing and robust update to ensure convergence and validity.
• Score: 19.853943355530017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computerized Adaptive Testing (CAT), as a key technology for personalized education, aims to accurately assess examinee proficiency by retrieving exercises dynamically matching current ability estimates. However, existing CAT research is constrained by limitations of static offline data and isolated component optimization. Restricted by partial labels in offline logs, researchers degrade the dynamic assessment process into static sequence prediction. Current research focuses on isolated perspectives, e.g., selection or diagnosis, neglecting the overall CAT interaction process. To address this, we propose AgentCAT, a Large Language Model-based multi-agent simulation system, to construct a high-fidelity benchmarking environment for dynamic testing. This framework comprises three modules: (1) The examinee agent with memory retrieval and Chain-of-Thought reasoning simulates responses based on cognitive profiles; (2) The selection agent uses coarse-to-fine bucketing and knowledge graph exploration to balance local difficulty and global coverage; (3) The supervisor uses dual-auditing and robust update to ensure convergence and validity. To validate the framework, we evaluated on two real-world datasets across three dimensions: macro-level ability convergence, micro-level interaction logic, and data sparsity resilience. Results show AgentCAT achieves effective ability estimation, and its selection strategy balances difficulty adaptation and instructional coherence, aligning with human pedagogical intuition.

Related papers

• CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents [1.4057557845067372]
  CIPHER is an automated data science agent that leverages test-time scaling through the generation and selection of multiple initial states for concurrent execution.<n>We demonstrate that CIPHER exceeds state-of-the-art performance in matched-model comparisons.<n>Our empirical study characterizes the design space of the Decoupled Exploration-Selection framework.
  arXiv  Detail & Related papers  (2026-07-15T22:01:21Z)
• From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning [12.024430772980502]
  We introduce an agent-centric benchmarking paradigm for evaluating large language models.<n>A teacher agent generates candidate problems, an orchestrator agent rigorously verifies their validity and guards against adversarial attacks.<n>If a student correctly solves the problem, the orchestrator prompts the teacher to generate more challenging variants.
  arXiv  Detail & Related papers  (2026-02-27T06:54:32Z)
• gencat: Generative computerized adaptive testing [1.0162911785128765]
  We propose GENCAT, a novel CAT framework that leverages Large Language Models for knowledge estimate and question selection.<n>First, we develop a Generative Item Response Theory (GIRT) model that enables us to estimate student knowledge from their open-ended responses and predict responses to unseen questions.<n>Second, we introduce three question selection algorithms that leverage the generative capabilities of the GIRT model, based on the uncertainty, linguistic diversity, and information of sampled student responses.
  arXiv  Detail & Related papers  (2026-02-23T16:28:46Z)
• Adaptive Causal Coordination Detection for Social Media: A Memory-Guided Framework with Semi-Supervised Learning [1.5683405037750644]
  ACCD adopts a three-stage, progressive architecture that leverages a memory-guided adaptive mechanism to learn and retain optimal detection configurations.<n>We conduct a comprehensive evaluation using real-world datasets, including the Twitter IRA dataset, Reddit coordination traces, and several widely-adopted bot detection benchmarks.<n>ACCD achieves an F1-score of 87.3% in coordinated attack detection, representing a 15.2% improvement over the strongest existing baseline.
  arXiv  Detail & Related papers  (2026-01-01T17:27:52Z)
• Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback [51.22403664895878]
  Agent2World is a tool-augmented multi-agent framework that achieves strong inference-time world-model generation.<n>It also serves as a data engine for supervised fine-tuning, by grounding generation in multi-agent feedback.
  arXiv  Detail & Related papers  (2025-12-26T18:54:14Z)
• AgenticIQA: An Agentic Framework for Adaptive and Interpretable Image Quality Assessment [69.06977852423564]
  Image quality assessment (IQA) reflects both the quantification and interpretation of perceptual quality rooted in the human visual system.<n>AgenticIQA decomposes IQA into four subtasks -- distortion detection, distortion analysis, tool selection, and tool execution.<n>To support training and evaluation, we introduce AgenticIQA-200K, a large-scale instruction dataset tailored for IQA agents, and AgenticIQA-Eval, the first benchmark for assessing the planning, execution, and summarization capabilities of VLM-based IQA agents.
  arXiv  Detail & Related papers  (2025-09-30T09:37:01Z)
• Creative Adversarial Testing (CAT): A Novel Framework for Evaluating Goal-Oriented Agentic AI Systems [0.0]
  Creative Adversarial Testing (CAT) is a novel approach designed to capture and analyze the complex relationship between Agentic AI tasks and the system's intended objectives.<n>We validate the CAT framework through extensive simulation using synthetic interaction data modeled after Alexa+ audio services.<n>Our results demonstrate that the CAT framework provides unprecedented insights into goal-task alignment, enabling more effective optimization and development of Agentic AI systems.
  arXiv  Detail & Related papers  (2025-09-26T23:52:20Z)
• Towards Interactive Lesion Segmentation in Whole-Body PET/CT with Promptable Models [9.254535645473311]
  We present our submission to the autoPET/CT IV challenge.<n>We extend the framework with promptable capabilities by encoding user-provided foreground and background clicks as additional input channels.<n>Our ensemble of EDT-based models, trained with and without external data, achieves the strongest cross-validation performance.
  arXiv  Detail & Related papers  (2025-08-29T14:49:58Z)
• Topology-Aware Modeling for Unsupervised Simulation-to-Reality Point Cloud Recognition [63.55828203989405]
  We introduce a novel Topology-Aware Modeling (TAM) framework for Sim2Real UDA on object point clouds.<n>Our approach mitigates the domain gap by leveraging global spatial topology, characterized by low-level, high-frequency 3D structures.<n>We propose an advanced self-training strategy that combines cross-domain contrastive learning with self-training.
  arXiv  Detail & Related papers  (2025-06-26T11:53:59Z)
• Survey of Computerized Adaptive Testing: A Machine Learning Perspective [66.26687542572974]
  Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees. This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method.
  arXiv  Detail & Related papers  (2024-03-31T15:09:47Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Quality meets Diversity: A Model-Agnostic Framework for Computerized Adaptive Testing [60.38182654847399]
  Computerized Adaptive Testing (CAT) is emerging as a promising testing application in many scenarios. We propose a novel framework, namely Model-Agnostic Adaptive Testing (MAAT) for CAT solution.
  arXiv  Detail & Related papers  (2021-01-15T06:48:50Z)
• Dynamic Federated Learning [57.14673504239551]
  Federated learning has emerged as an umbrella term for centralized coordination strategies in multi-agent environments. We consider a federated learning model where at every iteration, a random subset of available agents perform local updates based on their data. Under a non-stationary random walk model on the true minimizer for the aggregate optimization problem, we establish that the performance of the architecture is determined by three factors, namely, the data variability at each agent, the model variability across all agents, and a tracking term that is inversely proportional to the learning rate of the algorithm.
  arXiv  Detail & Related papers  (2020-02-20T15:00:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.