Related papers: From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints

From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints

URL: http://arxiv.org/abs/2605.28483v1
Date: Wed, 27 May 2026 13:41:10 GMT
Title: From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints
Abstract summary: We present an end-to-end alignment pipeline that uses a large language model (LLM) as a constrained, evidence-producing tagger.<n>We evaluate our approach on a dataset built from the Computer Science department's competency referential at the Université de Technologie de Compigne.
Score: 26.702635306128865
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Linking learning resources to a structured competency framework is key to enabling competency-based search and curriculum analytics in Learning Management Systems (LMS). However, manual tagging is labor-intensive, and fully automatic methods often lack transparency. In this paper, we present an end-to-end alignment pipeline that uses a large language model (LLM) as a constrained, evidence-producing tagger. LMS resources -both instructional content and assessments -are first segmented into meaningful pedagogical fragments. For each fragment, a small set of candidate competencies is retrieved from structured competency profiles enriched with graph-based context. The LLM then selects the most relevant competencies from this set and provides supporting evidence spans from the fragment text. These predictions are refined using the structure of the competency graph and aggregated at the resource level. We evaluate our approach on a dataset built from the Computer Science department's competency referential at the Université de Technologie de Compiègne (UTC), covering 22 competencies across multiple course materials. Our LLM+BM25+Graph (LBG) pipeline achieves strong results, with a micro-F1 of 0.57 and macro-F1 of 0.50 at the fragment level, 0.51 macro-F1 at the resource level, and an MRR of 0.82outperforming zero-shot and few-shot LLM variants, retrieval/similarity baselines, and supervised classifiers -while also producing more mechanically traceable evidence spans to support human auditing and educational analysis.

Related papers

KG-First, LLM-Fallback: A Hybrid Microservice for Grounded Skill Search and Explanation [26.702635306128865]
Authoritative competency frameworks such as ESCO, ROME, and O*NET are essential for aligning education with labor market needs, yet their technical complexity and structural heterogeneity hinder practical adoption by educators.<n>This paper introduces SkillGraph-Service, a microservice designed to bridge this gap by unifying these resources into a provenance-preserving Knowledge Graph (KG)<n>The resulting architecture offers a practical, scalable, and auditable solution for integrating complex skill data into digital learning ecosystems.
arXiv Detail & Related papers (2026-05-02T19:07:56Z)
Focus, Merge, Rank: Improved Question Answering Based on Semi-structured Knowledge Bases [2.6524539020042663]
We present FocusedRetriever, a modular SKB-based framework for multi-hop question answering.<n>It integrates components (VSS-based entity search, LLM-based generation of Cypher queries and pairwise re-ranking) in a way that enables it to outperform state-of-the-art methods.<n>The average first-hit rate exceeds that of the second-best method by 25.7%.
arXiv Detail & Related papers (2025-05-14T09:35:56Z)
SCAN: Structured Capability Assessment and Navigation for LLMs [54.54085382131134]
textbfSCAN (Structured Capability Assessment and Navigation) is a practical framework that enables detailed characterization of Large Language Models.<n>SCAN incorporates four key components:.<n>TaxBuilder, which extracts capability-indicating tags from queries to construct a hierarchical taxonomy;.<n>RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag;.<n>A PC$2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach achieves significantly higher accuracy compared to classic LLM-as-a-Judge method
arXiv Detail & Related papers (2025-05-10T16:52:40Z)
From Course to Skill: Evaluating LLM Performance in Curricular Analytics [2.5104969073405976]
Large language models (LLMs) are promising for handling large-scale, unstructured curriculum data.<n>We systematically evaluate four text alignment strategies based on LLMs or traditional NLP methods for skill extraction.<n>Our findings highlight the promise of LLMs in analyzing brief and abstract curriculum documents, but also reveal that their performance can vary significantly depending on model selection and prompting strategies.
arXiv Detail & Related papers (2025-05-05T02:46:23Z)
Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
Instruct-LF is a goal-oriented latent factor discovery system.<n>It integrates instruction-following ability with statistical models to handle noisy datasets.
arXiv Detail & Related papers (2025-02-21T02:03:08Z)
KcMF: A Knowledge-compliant Framework for Schema and Entity Matching with Fine-tuning-free LLMs [14.376057807754668]
Large language models (LLMs) suffer from hallucinations and confusion about task instructions.<n>This study presents the Knowledge-Compliant Matching Framework (KcMF) that addresses these issues without the need for domain-specific fine-tuning.
arXiv Detail & Related papers (2024-10-16T11:50:02Z)
Integrating Planning into Single-Turn Long-Form Text Generation [66.08871753377055]
We propose to use planning to generate long form content. Our main novelty lies in a single auxiliary task that does not require multiple rounds of prompting or planning. Our experiments demonstrate on two datasets from different domains, that LLMs fine-tuned with the auxiliary task generate higher quality documents.
arXiv Detail & Related papers (2024-10-08T17:02:40Z)
How to Make LLMs Strong Node Classifiers? [70.14063765424012]
Language Models (LMs) are challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs)<n>We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks.
arXiv Detail & Related papers (2024-10-03T08:27:54Z)
Text Clustering as Classification with LLMs [9.128151647718251]
We propose a novel framework that reframes text clustering as a classification task by harnessing the in-context learning capabilities of Large Language Models.<n>By leveraging the advanced natural language understanding and generalization capabilities of LLMs, the proposed approach enables effective clustering with minimal human intervention.<n> Experimental results on diverse datasets demonstrate that our framework achieves comparable or superior performance to state-of-the-art embedding-based clustering techniques.
arXiv Detail & Related papers (2024-09-30T16:57:34Z)
FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability [70.84333325049123]
FoFo is a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats. This paper presents FoFo, a pioneering benchmark for evaluating large language models' (LLMs) ability to follow complex, domain-specific formats.
arXiv Detail & Related papers (2024-02-28T19:23:27Z)
Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
arXiv Detail & Related papers (2023-05-31T03:18:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.