Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment

• URL: http://arxiv.org/abs/2607.16202v1
• Date: Tue, 05 May 2026 02:36:09 GMT
• Title: Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment
• Abstract summary: We evaluate nine open-weight language models between 135M and 3B parameters on a benchmark designed for structured local deployment.<n>A shared parameter-efficient fine-tuning pipeline adapts a subset of models using 4-bit NF4 quantization with DoRA/LoRA-style adapters on an NVIDIA L4-class budget.<n>In base evaluation, Qwen Coder 3B leads at 75.67% strict accuracy, followed by Qwen2.5 1.5B at 67.10%, Qwen3.5 2B at 64.98%, and Granite 3.3 2B at 64.61%.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI democratization is not primarily a question of matching frontier-scale generality; it is a question of whether capable models can be selected, audited, and specialized under hardware and governance constraints that ordinary institutions can actually satisfy. This paper studies that problem through a controlled evaluation of nine open-weight language models between 135M and 3B parameters on a 1,085-example, 16-topic multiple-choice benchmark designed for structured local deployment. The benchmark emphasizes symbolic precision, constrained formatting, extraction, and short-horizon semantic decision making under a strict one-letter output protocol. A shared parameter-efficient fine-tuning pipeline then adapts a subset of models using 4-bit NF4 quantization with DoRA/LoRA-style adapters on an NVIDIA L4-class budget. In base evaluation, Qwen Coder 3B leads at 75.67% strict accuracy, followed by Qwen2.5 1.5B at 67.10%, Qwen3.5 2B at 64.98%, and Granite 3.3 2B at 64.61%. On the shared 108-example held-out fine-tuning split, adaptation improves Qwen Coder 3B by +26.85 points, SmolLM2 1.7B by +25.92, Qwen2.5 1.5B by +19.44, SmolLM2 360M by +10.18, and SmolLM2 135M by +5.55. Across ranking, topic-level heterogeneity, difficulty strata, failure composition, efficiency frontiers, and topic-conditioned transfer, the same conclusion recurs: a disciplined workflow of benchmark construction, cross-model evaluation, and low-cost specialization already makes a subset of sub-3B models viable as local experts for structured niche workloads.

Related papers

• State of Thought Enables Endogenous Reasoning [48.902117018115256]
  Test-time reasoning has emerged as a major approach to improving the capabilities of Large Language Models (LLMs)<n>We propose State of Thought, a new reasoning paradigm that enables endogenous reasoning in LLMs.<n>We show that SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%.
  arXiv  Detail & Related papers  (2026-09-13T05:55:03Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models [2.1665689529884697]
  We present B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM.<n>B1ade-embed, a 335M parameter retrieval model constructed via parameter-free fusion of five pretrained encoders achieves top MTEB scores among sub-500M models with zero additional training.<n>B1ade-1B cites retrieved passages in 42.4% of responses, exceeding the attribution rate of its training distribution by 5.5 percentage points.
  arXiv  Detail & Related papers  (2026-07-29T22:41:25Z)
• Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data [6.3745740668603075]
  Large language models (LLMs) for domain-specific structured evaluation tasks incurs prohibitive latency, cost, and data-privacy overhead.<n>We present a hybrid framework that fine-tunes a small language model on only 219 curated examples and couples it with a deterministic rule-based postprocessing layer.<n>Our system achieves 100% structural validity, 83.0% human-validated overall accuracy, and 100% accuracy on the most critical classification field in blind evaluation on 53 unseen production transcripts.
  arXiv  Detail & Related papers  (2026-06-04T07:09:37Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models [0.0]
  We study whether structured synthetic reasoning data can improve arithmetic reasoning in small language models.<n>We generate a 21,250-example corpus of grade-school arithmetic word-problem variants using GPT-5-mini.<n>Fine-tuned models produce shorter reasoning traces, make fewer arithmetic and distractor-use errors, and benefit more consistently from self-consistency sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:29Z)
• Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization [54.749573452394664]
  We find that accuracy collapses as instance data grows, even when the formulation itself is simple.<n>We address this via a simple inference-time approach, BIND, which externalizes numeric data to structured files so the model binds data rather than from prompt prompt.<n>We validate our hypothesis by finetuning a model exclusively on binding and show that it outperforms end-to-end SFT and RL across three structurally distinct optimization categories.
  arXiv  Detail & Related papers  (2026-05-20T21:25:41Z)
• LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening [69.1037790901185]
  We present LLMEval-Logic, a Chinese logical reasoning benchmark built from realistic situational scenarios.<n>Its pipeline forward-authors and expert-audits natural-language items together with their reference formalizations, verifies annotated answers with Z3, constructs expert rubrics for natural-to-formal grading, and hardens selected items through a closed-loop adversarial workflow.
  arXiv  Detail & Related papers  (2026-05-19T09:40:29Z)
• Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment [0.05586191108738562]
  Small language models (SLMs) have sufficient reasoning capability for sub-second, zero-marginal-cost, self-hosted task classification.<n>Study 1 is a harmonized offline benchmark of Phi-3.5-mini, Qwen2.5-1.5B, and Qwen-2.5-3B on identical Azure T4 hardware, serving stack, quantization, and a fixed 60-case corpus.<n>Study 2 is a pre-registered four-arm randomized experiment under synthetic traffic with an effective sample size of 60 unique cases per arm.
  arXiv  Detail & Related papers  (2026-03-26T15:57:46Z)
• How Small Can 6G Reason? Scaling Tiny Language Models for AI-Native Networks [3.099103925863002]
  We study the scaling behavior and deployment efficiency of compact language models for network-level semantic reasoning in AI-native 6G systems.<n>We evaluate models ranging from 135M (SmolLM2-135M) to 7B parameters (Qwen2.5-7B), including mid-scale architectures such as Llama-3.2-1B, Granite-1B, and Qwen2.5-3B.
  arXiv  Detail & Related papers  (2026-03-02T18:19:49Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks [0.0]
  Small or compact models, though more efficient, often lack sufficient support for underrepresented languages.<n>This work explores the potential of parameter-efficient fine-tuning of compact open-weight language models to handle reasoning-intensive tasks.<n> tuning method with joint task topic and step-by-step solution generation outperforms standard chain-of-thought tuning in matching tasks.
  arXiv  Detail & Related papers  (2025-03-18T07:44:49Z)
• ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization [73.60493264901359]
  We present a unified framework for rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings.<n>We show that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off.<n>Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.
  arXiv  Detail & Related papers  (2025-02-04T18:59:26Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.