Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks

• URL: http://arxiv.org/abs/2607.03801v1
• Date: Sat, 04 Jul 2026 10:09:32 GMT
• Title: Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks
• Abstract summary: We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices.<n>We compare three LoRA-based fine-tuning paradigms on a unified setup across several Qwen3 models.<n> TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4.
• Score: 0.11458853556386796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices. We study how to adapt them for and use them on verifiable multiple-choice tasks. We compare three LoRA-based fine-tuning paradigms (label generation, gold only, and our discriminative classification head) on a unified setup across several Qwen3 models from 0.6B to 8B and five benchmarks: HellaSwag, WinoGrande, PIQA, SciQ and ARC-C. Classification-head fine-tuning reliably outperforms label generation (+2-3%) at the 0.6B and 1.7B scales. Further, TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4. The performance we report for Qwen3-0.6B and Qwen3-1.7B are SOTA on HellaSwag, WinoGrande, and PIQA.

Related papers

• Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction [5.070542698701158]
  Large language models (LLMs) achieve strong relation extraction (RE)<n>We investigate how far small language models (SLMs) can close this gap across general-domain and literary text.
  arXiv  Detail & Related papers  (2026-06-21T17:24:31Z)
• Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit [5.437038443632133]
  We test an implicit assumption that scale and general capability are sufficient for nuanced classification of misinformation discourse.<n>We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs, and fine-tuned DistilBERT and RoBERTa.<n>In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice.
  arXiv  Detail & Related papers  (2026-06-02T22:58:59Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• Deterministic Fuzzy Triage for Legal Compliance Classification and Evidence Retrieval [0.0]
  Legal teams increasingly use machine learning to triage large volumes of contractual evidence.<n>Many models are opaque, non-deterministic, and difficult to align with frameworks such as HIPAA or NERC-CIP.<n>We study a simple, reproducible alternative based on deterministic dual encoders and transparent fuzzy triage bands.
  arXiv  Detail & Related papers  (2026-03-08T00:31:34Z)
• Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderation [0.4349640169711269]
  Bielik Guard is a family of compact Polish language safety classifiers.<n>Fine-tuned on a community-annotated dataset of 6,885 Polish texts, these models classify content across five safety categories: Hate/Aggression, Vulgarities, Sexual Content, Crime, and Self-Harm.
  arXiv  Detail & Related papers  (2026-02-08T12:57:04Z)
• CLaC at DISRPT 2025: Hierarchical Adapters for Cross-Framework Multi-lingual Discourse Relation Classification [0.0509780930114934]
  Task 3 introduces a unified set of 17 discourse relation labels across 39 corpora in 16 languages and six discourse frameworks.<n>We first benchmark the task by fine-tuning multilingual BERT-based models with two argument-ordering strategies and progressive unfreezing ratios.<n>We then evaluate prompt-based large language models in zero-shot and few-shot settings to understand how LLMs respond to the newly proposed unified labels.
  arXiv  Detail & Related papers  (2025-09-21T03:34:31Z)
• Self-Play Preference Optimization for Language Model Alignment [75.83359213697854]
  Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences. We propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium.
  arXiv  Detail & Related papers  (2024-05-01T17:59:20Z)
• InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT [75.29359361404073]
  InheritSumm is a versatile and compact summarization model derived from GPT-3.5 through distillation. It achieves similar or superior performance to GPT-3.5 in zeroshot and fewshot settings.
  arXiv  Detail & Related papers  (2023-05-22T14:52:32Z)
• Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification [58.720142291102135]
  This case study investigates the task of job classification in a real-world setting. The goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position.
  arXiv  Detail & Related papers  (2023-03-13T14:09:53Z)
• Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases [55.45617404586874]
  We propose a few-shot instruction-based method for prompting pre-trained language models (LMs) We show that large LMs can detect different types of fine-grained biases with similar and sometimes superior accuracy to fine-tuned models.
  arXiv  Detail & Related papers  (2021-12-15T04:19:52Z)
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing [117.41016786835452]
  This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model. vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. We propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics.
  arXiv  Detail & Related papers  (2021-11-18T06:48:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.