Multi-Model Synthetic Training for Mission-Critical Small Language Models

• URL: http://arxiv.org/abs/2509.13047v1
• Date: Tue, 16 Sep 2025 13:04:48 GMT
• Title: Multi-Model Synthetic Training for Mission-Critical Small Language Models
• Authors: Nolan Platt, Pragyansmita Nayak,
• Abstract summary: We present a novel approach that achieves a 261x cost reduction for maritime intelligence.<n>Our method transforms 3.2 billion Automatic Identification System (AIS) vessel tracking records into 21,543 synthetic question and answer pairs.<n>The resulting fine-tuned Qwen2.5-7B model achieves 75% accuracy on maritime tasks, while being substantially cheaper than using a larger model for inference.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across many domains, yet their appli- cation to specialized fields remains constrained by the scarcity and complexity of domain-specific training data. We present a novel approach that achieves a 261x cost reduction for maritime intelligence by using LLMs as one-time teachers rather than using them directly for inference. Our method transforms 3.2 billion Automatic Identification System (AIS) vessel tracking records into 21,543 synthetic question and answer pairs through multi-model generation (GPT-4o and o3-mini), preventing over- fitting and ensuring accurate reasoning. The resulting fine-tuned Qwen2.5-7B model achieves 75% accuracy on maritime tasks, while being substantially cheaper than using a larger model for inference. We show that smaller, cheaper models - when fine tuned properly - can provide similar accuracy compared to larger models that are prohibitively expensive. Our work contributes to the growing field of synthetic dataset generation for specialized AI applications and presents a highly reproducible framework for domains where manual annotation is infeasible. Beyond expand- ing research in the growing field of specialized small language models, our approach has immediate applications in maritime safety, security operations, and vessel traffic management systems in various industries.

Related papers

• Transport and Merge: Cross-Architecture Merging for Large Language Models [59.53629883788284]
  Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data.<n>Many real-world deployments rely on smaller models trained or adapted from low-resource data.<n>This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets.
  arXiv  Detail & Related papers  (2026-02-05T09:57:57Z)
• Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models [78.73992315826035]
  We introduce Youtu-LLM, a lightweight language model that harmonizes high computational efficiency with native agentic intelligence.<n>Youtu-LLM is pre-trained from scratch to systematically cultivate reasoning and planning capabilities.
  arXiv  Detail & Related papers  (2025-12-31T04:25:11Z)
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs [80.72350166388601]
  Nemotron Elastic is a framework for building reasoning-oriented LLMs.<n>It embeds nested submodels within a single parent model.<n>Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment.
  arXiv  Detail & Related papers  (2025-11-20T18:59:21Z)
• Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning [20.41220110321494]
  We propose Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning.<n>STEER is a domain-agnostic framework that performs fine-grained, step-level routing between smaller and larger language models.<n>Our results establish model-internal confidence as a robust, domain-agnostic signal for model routing.
  arXiv  Detail & Related papers  (2025-11-09T02:33:08Z)
• Revealing the Power of Post-Training for Small Language Models via Knowledge Distillation [43.68215777330875]
  We introduce a systematic post-training pipeline that efficiently enhances small model accuracy.<n>The resulting instruction-tuned model achieves state-of-the-art performance.<n>This work provides a practical and efficient solution for developing high-performance language models on Ascend edge devices.
  arXiv  Detail & Related papers  (2025-09-30T16:40:55Z)
• Case Studies of Generative Machine Learning Models for Dynamical Systems [0.5499796332553707]
  This paper focuses on two case studies of optimally controlled systems that are commonly understood and employed in aircraft guidance.<n>We report GAIMs that are trained with a relatively small set, of the order of a few hundred, of examples and with underlying governing equations.<n>New models are able to synthesize data that satisfy the governing equations and are statistically similar to the training data despite small volumes of training data.
  arXiv  Detail & Related papers  (2025-08-06T13:59:14Z)
• EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models [64.18350535770357]
  We propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning.<n>Our approach only leverages a small number of samples to search for the desired pruning policy.<n>We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering.
  arXiv  Detail & Related papers  (2025-03-19T16:07:04Z)
• Lightweight Multimodal Artificial Intelligence Framework for Maritime Multi-Scene Recognition [5.667043618885205]
  Maritime Multi-Scene Recognition is crucial for enhancing the capabilities of intelligent marine robotics.<n>Our framework integrates image data, textual descriptions and classification vectors generated by a Multimodal Large Language Model (MLLM)<n>Our model achieves 98$%$ accuracy, surpassing previous SOTA models by 3.5$%$.<n>This work provides a high-performance solution for real-time maritime scene recognition, enabling Autonomous Surface Vehicles (ASVs) to support environmental monitoring and disaster response in resource-limited settings.
  arXiv  Detail & Related papers  (2025-03-10T06:47:38Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• A Practical Approach to Underwater Depth and Surface Normals Estimation [3.0516727053033392]
  This paper presents a novel deep learning model for Monocular Depth and Surface Normals Estimation (MDSNE)<n>It is specifically tailored for underwater environments, using a hybrid architecture that integrates CNNs with Transformers.<n>Our model reduces parameters by 90% and training costs by 80%, allowing real-time 3D perception on resource-constrained devices.
  arXiv  Detail & Related papers  (2024-10-02T22:41:12Z)
• Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning [0.9787137564521711]
  We show how a combination of semantic search, prompt engineering, and fine-tuning can be applied to dramatically improve the ability of LLMs to execute these tasks accurately. We compare both proprietary models, like GPT-4, and open-source models, like Llama-2-70b, as well as various embedding methods.
  arXiv  Detail & Related papers  (2024-04-16T03:39:16Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• BYOM: Building Your Own Multi-Task Model For Free [69.63765907216442]
  BYOM-FFT is for merging fully finetuned models, while BYOM-LoRA is for LoRA-finetuned models. Experiments on computer vision and natural language processing tasks show that the proposed BYOM methods outperform existing merging methods by a large margin.
  arXiv  Detail & Related papers  (2023-10-03T08:39:33Z)
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes [91.58845026796149]
  We introduce Distilling step-by-step, a new mechanism that trains small models that outperform large language models. We present three findings across 4 NLP benchmarks.
  arXiv  Detail & Related papers  (2023-05-03T17:50:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.