Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment

• URL: http://arxiv.org/abs/2609.07370v1
• Date: Mon, 07 Sep 2026 11:44:38 GMT
• Title: Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment
• Abstract summary: Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents.<n>Model Context Protocol (MCP)-style tool invocation demands more than fluent generation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions.

Related papers

• ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
  Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
  arXiv  Detail & Related papers  (2026-09-15T04:59:03Z)
• Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment [4.582004238093247]
  Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation.<n>We deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation.<n>We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition.
  arXiv  Detail & Related papers  (2026-08-16T11:52:13Z)
• Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models [0.0]
  We investigate Parishad, a structured multi-agent system involving five roles, by deploying it on Qwen2.5-7B-Instruct.<n>Our results demonstrate that communication format and implementation details determine outcomes more than architectural complexity.
  arXiv  Detail & Related papers  (2026-07-29T13:53:02Z)
• When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More [10.283803346813047]
  A growing line of work equips large language model (LLM) agents with graph neural networks (GNNs) as callable tools.<n>We expose a frozen GNN to a ReAct-style LLM agent as an explicit tool and measure, on node classification over a text-attributed graph.<n>We find the agent does not exercise judgment: its predictions agree with the raw GNN's 97.6-99.2% of the time (5 seeds)<n>We find the cost of deference does not shrink as capability grows and grows where alternatives emerge.
  arXiv  Detail & Related papers  (2026-06-12T14:13:19Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models [0.5482532589225553]
  This paper targets on-device and low-cost small language model (SLM) deployments.<n>We show that hard output constraints improve reliability without changing the underlying answer.<n>We introduce emphconstraint tax, a measurement protocol for isolating the answer and executable-accuracy loss.
  arXiv  Detail & Related papers  (2026-05-20T07:11:32Z)
• Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling [64.40340291543971]
  We propose Speculative Interaction Agents to enable real-time interaction with small edge-scale models.<n>We also present a clock-based training methodology that adapts the model to handle streaming inputs and asynchronous responses.<n>This approach provides 1.6-2.2$times$ speedups with the Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct models across multiple tool calling benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T11:20:52Z)
• LLM Agents Already Know When to Call Tools -- Even Without Reasoning [25.40369702634587]
  LLM agents tend to call tools indiscriminately, even when the model can answer directly.<n>No existing benchmark systematically studies when a tool call is actually needed.<n>We propose When2Tool, a benchmark of 18 environments spanning three categories of tool necessity.
  arXiv  Detail & Related papers  (2026-05-10T01:37:40Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm [60.14048367611333]
  MonkeyOCR is a vision-language model for document parsing.<n>It advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm.
  arXiv  Detail & Related papers  (2025-06-05T16:34:57Z)
• s1: Simple test-time scaling [148.4204982041058]
  Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance.<n>We seek the simplest approach to achieve test-time scaling and strong reasoning performance.
  arXiv  Detail & Related papers  (2025-01-31T18:48:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.