From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions

• URL: http://arxiv.org/abs/2606.18257v1
• Date: Wed, 06 May 2026 02:21:57 GMT
• Title: From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions
• Abstract summary: This work evaluates six widely-used LLMs through a Bloom's taxonomy lens.<n>We generate and analyze 20,700 questions across computer science, K--12 math, and social-science domains.
• Score: 42.08829966803665
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While LLMs show promise in automating educational content creation, their ability to generate questions that stimulate higher-order thinking remains understudied. This work evaluates six widely-used LLMs through a Bloom's Taxonomy lens, focusing on their capacity to transcend rote memorization and achieve cognitive leaps. Using a hybrid human--AI evaluation protocol, we generate and analyze 20{,}700 questions across computer science, K--12 math, and social-science domains. Key contributions include: (1) a fine-grained prompting strategy that reduces question repetitiveness by 24.45\% for Qwen2.5-7B-Instruct, and increases the proportion of higher-order cognitive level outputs by 11.53\% for InternLM3-8B-Instruct; (2) quantitative metrics for cognitive shift intensity (CogShift) and category drift, revealing InternLM3's superior performance in multi-level transitions; (3) an interpretability analysis revealing metric-level correlations that enhance the transparency of Chain-of-Thought prompting. Our findings highlight the importance of cognitive-aware prompt design and provide benchmarks for deploying LLMs in personalized learning systems.

Related papers

• UniCog: Uncovering Cognitive Abilities of LLMs through Latent Mind Space Analysis [69.50752734049985]
  A growing body of research suggests that the cognitive processes of large language models (LLMs) differ fundamentally from those of humans.<n>We propose UniCog, a unified framework that analyzes LLM cognition via a latent mind space.
  arXiv  Detail & Related papers  (2026-01-25T16:19:00Z)
• SpatialTree: How Spatial Abilities Branch Out in MLLMs [109.32057088014942]
  We introduce a cognitive-science-inspired hierarchy that organizes spatial abilities into four levels: low-level perception (L1), mental mapping (L2), simulation (L3), and agentic competence (L4)<n>We find that naive RL that encourages extensive "thinking" is unreliable: it helps complex reasoning but hurts intuitive perception.<n>We propose a simple auto-think strategy that suppresses unnecessary deliberation, enabling RL to consistently improve performance across all levels.
  arXiv  Detail & Related papers  (2025-12-23T18:59:46Z)
• SkillGen: Learning Domain Skills for In-Context Sequential Decision Making [24.41349550520032]
  We introduce SkillGen, a skill-based ICL framework for structured sequential reasoning.<n>We show that SkillGen achieves consistent gains, improving progress rate by 5.9%-16.5% on average across models.
  arXiv  Detail & Related papers  (2025-11-18T17:09:21Z)
• Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting [11.725875396424927]
  We introduce Attention-Shifting (AS) framework for selective unlearning.<n>AS is driven by two design objectives: (1) context-preserving suppression that attenuates attention to fact-bearing tokens without disrupting LLMs' linguistic structure; and (2) hallucination-resistant response shaping that discourages fabricated completions when queried about unlearning content.<n> Experimental results show that AS improves performance over the state-of-the-art unlearning methods, achieving up to 15% higher accuracy on the ToFU benchmark and 10% on the TDEC benchmark, while maintaining competitive hallucination-free unlearning effectiveness.
  arXiv  Detail & Related papers  (2025-10-20T06:50:03Z)
• Beyond Scale: Small Language Models are Comparable to GPT-4 in Mental Health Understanding [12.703061322251093]
  Small Language Models (SLMs) are privacy-preserving alternatives to Large Language Models (LLMs)<n>This paper investigates the mental health understanding capabilities of current SLMs through systematic evaluation across classification tasks.<n>Our work highlights the potential of SLMs in mental health understanding, showing they can be effective privacy-preserving tools for analyzing sensitive online text data.
  arXiv  Detail & Related papers  (2025-07-09T02:40:02Z)
• Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation [106.17986469245302]
  Large language models (LLMs) have demonstrated impressive reasoning capacities that mirror human-like thinking.<n>Existing reasoning benchmarks either focus on domain-specific knowledge (crystallized intelligence) or lack interpretability.<n>We propose DRE-Bench, a dynamic reasoning evaluation benchmark grounded in a hierarchical cognitive framework.
  arXiv  Detail & Related papers  (2025-06-03T09:01:08Z)
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [93.8285345915925]
  Reasoning is a fundamental cognitive process that enables logical inference, problem-solving, and decision-making.<n>With the rapid advancement of large language models (LLMs), reasoning has emerged as a key capability that distinguishes advanced AI systems.<n>We categorize existing methods along two dimensions: (1) Regimes, which define the stage at which reasoning is achieved; and (2) Architectures, which determine the components involved in the reasoning process.
  arXiv  Detail & Related papers  (2025-04-12T01:27:49Z)
• DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding [61.26026947423187]
  Human experts excel at fine-grained visual discrimination by leveraging domain knowledge to refine perceptual features.<n>Current Multimodal Large Language Models (MLLMs) struggle to integrate reasoning into visual perception.<n>We propose DeepPerception, an MLLM enhanced with cognitive visual perception capabilities.
  arXiv  Detail & Related papers  (2025-03-17T04:06:34Z)
• CogLM: Tracking Cognitive Development of Large Language Models [20.138831477848615]
  We construct a benchmark CogLM based on Piaget's Theory of Cognitive Development.<n>CogLM comprises 1,220 questions spanning 10 cognitive abilities crafted by more than 20 human experts.<n>We find that advanced LLMs have demonstrated human-like cognitive abilities, comparable to those of a 20-year-old human.
  arXiv  Detail & Related papers  (2024-08-17T09:49:40Z)
• Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning [37.22349652230841]
  Large language models (LLMs) enable in-context learning (ICL) by conditioning on a few labeled training examples as a text-based prompt. In this paper, we demonstrate that factual knowledge is imperative for the performance of ICL in three core facets. We introduce a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL.
  arXiv  Detail & Related papers  (2023-09-26T09:06:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.