SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations

• URL: http://arxiv.org/abs/2603.15372v1
• Date: Mon, 16 Mar 2026 14:48:53 GMT
• Title: SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations
• Abstract summary: General-purpose language model (LLM) agents execute telecom operations through real API interfaces, or do they require structured domain guidance?<n>We introduce SKILLS, a benchmark framework comprising 37 telecom operations scenarios spanning 8 TM Forum Open API domains.<n>Results across 5 open-weight model conditions and 185 scenario-runs show consistent skill lift across all models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As telecommunications operators accelerate adoption of AI-enabled automation, a practical question remains unresolved: can general-purpose large language model (LLM) agents reliably execute telecom operations workflows through real API interfaces, or do they require structured domain guidance? We introduce SKILLS (Structured Knowledge Injection for LLM-driven Service Lifecycle operations), a benchmark framework comprising 37 telecom operations scenarios spanning 8 TM Forum Open API domains (TMF620, TMF621, TMF622, TMF628, TMF629, TMF637, TMF639, TMF724). Each scenario is grounded in live mock API servers with seeded production-representative data, MCP tool interfaces, and deterministic evaluation rubrics combining response content checks, tool-call verification, and database state assertions. We evaluate open-weight models under two conditions: baseline (generic agent with tool access but no domain guidance) and with-skill (agent augmented with a portable SKILL.md document encoding workflow logic, API patterns, and business rules). Results across 5 open-weight model conditions and 185 scenario-runs show consistent skill lift across all models. MiniMax M2.5 leads (81.1% with-skill, +13.5pp), followed by Nemotron 120B (78.4%, +18.9pp), GLM-5 Turbo (78.4%, +5.4pp), and Seed 2.0 Lite (75.7%, +18.9pp).

Related papers

• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• MADE: Belief-Driven Dual-Agent Coordination for Autonomous Model Deployment [19.126846876843356]
  LLM-based agents struggle with domain-specific tasks, motivating the integration of external tools.<n>Open-source community offers vast array of AI models typically released as heterogeneous research artifacts.<n>Transforming these into ready-to-call APIs is costly and labor-intensive.<n>We introduce Model Automated Deployment Engine (MADE), a dual-agent coordination system.
  arXiv  Detail & Related papers  (2026-08-02T12:14:43Z)
• Environment-free Synthetic Data Generation for API-Calling Agents [68.95520739708304]
  Training API-calling large language model (LLM) agents demands massive amounts of high-quality trajectories.<n>We propose an environment-free synthetic data generation approach that leverages LLMs as on-the-fly digital world models.<n>We evaluate our approach on the challenging AppWorld and OfficeBench benchmarks, which include both information-retrieval and state-changing tasks.
  arXiv  Detail & Related papers  (2026-07-18T17:36:44Z)
• Knowledge Boundary Probing and Demand-Guided Intervention for LLM-Based Power System Code Generation [8.136015158683694]
  We show that first-pass failures in power-system code generation are dominated by structured API-knowledge boundary: hallucinated function names, misused parameters, and mishandled result tables.<n>We introduce PowerCodeBench, an execution-validated benchmark generator that pairs natural-language operator queries with pandapower code and numerical ground truth.<n>We also introduce an L0-L3 documentation-driven probing procedure that measures per-model API knowledge profiles.
  arXiv  Detail & Related papers  (2026-05-29T16:06:34Z)
• Mono2Sls: Automated Monolith-to-Serverless Migration via Multi-Stage Pipeline with Static Analysis [39.8879463570937]
  We present Mono2Sls, an automated pipeline that converts monolithic web backends into deployable AWS SAM applications.<n>Mono2Sls achieves 100% deployment success without manual fixes.<n>It also reaches 66.1% end-to-end correctness and 98.7% API-coverage F1.
  arXiv  Detail & Related papers  (2026-04-27T14:44:07Z)
• Interfaze: The Future of AI is built on Task-Specific Small Models [0.25489046505746704]
  We present Interfaze, a system that treats modern LLM applications as a problem of building and acting over context.<n>A thin controller sits on top of this stack and exposes a single, OpenAI-style endpoint.<n>On this architecture, Interfaze-Beta achieves 83.6% on MMLU-Pro, 91.4% on MMLU, 81.3% on GPQA-Diamond, 57.8% on LiveCodeBench v5, and 90.0% on AIME-2025.
  arXiv  Detail & Related papers  (2026-02-04T00:36:37Z)
• FARM: Field-Aware Resolution Model for Intelligent Trigger-Action Automation [0.0]
  We study the function-level configuration problem: generating complete applets with correct ingredient-to-field bindings.<n>We propose FARM (Field-Aware Resolution Model), a two-stage architecture for automated applet generation with full configuration.<n> FARM achieves 81% joint accuracy on Gold (62% Noisy, 70% One-shot) at the function level, where both trigger and action functions must match the ground truth.
  arXiv  Detail & Related papers  (2026-01-22T06:12:18Z)
• Beyond Rule-Based Workflows: An Information-Flow-Orchestrated Multi-Agents Paradigm via Agent-to-Agent Communication from CORAL [0.15199492741752027]
  We propose an Information-Flow-Orchestrated Multi-Agent Paradigm via Agent-to-Agent (A2A) Communication.<n>We evaluate our approach on the general-purpose benchmark GAIA, using the representative workflow-based MAS as the baseline.<n>Our method achieves 63.64% accuracy, outperforming OWL's 55.15% by 8.49 percentage points with comparable token consumption.
  arXiv  Detail & Related papers  (2026-01-14T21:35:51Z)
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models [194.64264251080454]
  We present GLM-4.5, an open-source Mixture-of-Experts (MoE) large language model with 355B total parameters and 32B activated parameters.<n>Through multi-stage training on 23T tokens and comprehensive post-training with expert model iteration and reinforcement learning, GLM-4.5 achieves strong performance across agentic, reasoning, and coding tasks.<n>We release both GLM-4.5 (355B parameters) and a compact version, GLM-4.5-Air (106B parameters), to advance research in reasoning and agentic AI systems.
  arXiv  Detail & Related papers  (2025-08-08T17:21:06Z)
• LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? [50.60770039016318]
  We present LiveMCPBench, the first comprehensive benchmark for benchmarking Model Context Protocol (MCP) agents.<n>LiveMCPBench consists of 95 real-world tasks grounded in the MCP ecosystem.<n>Our evaluation covers 10 leading models, with the best-performing model reaching a 78.95% success rate.
  arXiv  Detail & Related papers  (2025-08-03T14:36:42Z)
• MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models [76.72220653705679]
  We introduce MCPEval, an open-source framework that automates end-to-end task generation and deep evaluation of intelligent agents.<n> MCPEval standardizes metrics, seamlessly integrates with native agent tools, and eliminates manual effort in building evaluation pipelines.<n> Empirical results across five real-world domains show its effectiveness in revealing nuanced, domain-specific performance.
  arXiv  Detail & Related papers  (2025-07-17T05:46:27Z)
• ChatCFD: An LLM-Driven Agent for End-to-End CFD Automation with Domain-Specific Structured Reasoning [4.098524616768554]
  ChatCFD is an automated agent system for OpenFOAM simulations.<n>Its four-stage pipeline enables iterative trial-reflection-refinement for intricate setups.<n>ChatCFD shows strong potential as a modular component in MCP-based agent networks for collaborative multi-agent systems.
  arXiv  Detail & Related papers  (2025-05-28T08:43:49Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Model-driven realization of IDTA submodel specifications: The good, the bad, the incompatible? [49.60138105915326]
  Asset Administration Shells are trending in Industry 4.0. In February 2024, the Industrial Digital Twin Association announced 84 and released 18 AAS submodel specifications. We present a model-driven approach, which transforms extracted information from IDTA specifications into an intermediary meta-model and, from there, generates API code and tests.
  arXiv  Detail & Related papers  (2024-06-20T16:33:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.