Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

• URL: http://arxiv.org/abs/2512.05525v1
• Date: Fri, 05 Dec 2025 08:36:39 GMT
• Title: Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
• Authors: Nils Strassenburg, Boris Glavic, Tilmann Rabl,
• Abstract summary: Businesses increasingly rely on large language models (LLMs) to automate simple repetitive tasks.<n>LLMs require few, if any, training examples and can be utilized by users without expertise in model development.<n>We present our vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative.
• Score: 6.193292107042754
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Businesses increasingly rely on large language models (LLMs) to automate simple repetitive tasks instead of developing custom machine learning models. LLMs require few, if any, training examples and can be utilized by users without expertise in model development. However, this comes at the cost of substantially higher resource and energy consumption compared to smaller models, which often achieve similar predictive performance for simple tasks. In this paper, we present our vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative that performs well for this specific task. JITR retains the ease of use and low development effort of LLMs, while saving significant cost and energy. We discuss the main challenges in realizing our vision regarding the identification of recurring tasks and the creation of a custom model. Specifically, we argue that model search and transfer learning will play a crucial role in JITR to efficiently identify and fine-tune models for a recurring task. Using our JITR prototype Poodle, we achieve significant savings for exemplary tasks.

Related papers

• Evolution without Large Models: Training Language Model with Task Principles [52.44569608690695]
  A common training approach for language models involves using a large-scale language model to expand a human-provided dataset.<n>This method significantly reduces training costs by eliminating the need for extensive human data annotation.<n>However, it still faces challenges such as high carbon emissions during data augmentation and the risk of data leakage.
  arXiv  Detail & Related papers  (2025-07-08T13:52:45Z)
• GRAM: A Generative Foundation Reward Model for Reward Generalization [48.63394690265176]
  We develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning.<n>This model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning.
  arXiv  Detail & Related papers  (2025-06-17T04:34:27Z)
• Efficient LLM Collaboration via Planning [56.081879390960204]
  Small and large models take turns acting as planner and executor, exchanging plans in a multi-stage cascade to collaboratively solve tasks.<n>We demonstrate that COPE achieves performance comparable to large proprietary models, while drastically reducing the inference API cost.
  arXiv  Detail & Related papers  (2025-06-13T08:35:50Z)
• Intention-Conditioned Flow Occupancy Models [80.42634994902858]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones [12.265271319203794]
  We explore a technique for mapping parameters of a large Transformer to parameters of a smaller specialized model.<n>By making this transformation task-specific, we aim to capture a narrower scope of the knowledge needed for performing a specific task by a smaller model.<n>We study our method on image modeling tasks, showing that performance of generated models exceeds that of universal conditional models.
  arXiv  Detail & Related papers  (2025-06-06T00:05:38Z)
• ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation [35.160964210941955]
  We propose ModelGPT, a framework designed to determine and generate AI models tailored to the data or task descriptions provided by the user. Given user requirements, ModelGPT is able to provide tailored models at most 270x faster than the previous paradigms.
  arXiv  Detail & Related papers  (2024-02-18T11:24:34Z)
• BYOM: Building Your Own Multi-Task Model For Free [69.63765907216442]
  BYOM-FFT is for merging fully finetuned models, while BYOM-LoRA is for LoRA-finetuned models. Experiments on computer vision and natural language processing tasks show that the proposed BYOM methods outperform existing merging methods by a large margin.
  arXiv  Detail & Related papers  (2023-10-03T08:39:33Z)
• Simplified Temporal Consistency Reinforcement Learning [19.814047499837084]
  We show that a simple representation learning approach relying on a latent dynamics model trained by latent temporal consistency is sufficient for high-performance RL. Our approach outperforms model-free methods by a large margin and matches model-based methods' sample efficiency while training 2.4 times faster.
  arXiv  Detail & Related papers  (2023-06-15T19:37:43Z)
• Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning [65.268245109828]
  In data-rich domains such as vision, language, and speech, deep learning prevails to deliver high-performance task-specific models. Deep learning in resource-limited domains still faces multiple challenges including (i) limited data, (ii) constrained model development cost, and (iii) lack of adequate pre-trained models for effective finetuning. Model reprogramming enables resource-efficient cross-domain machine learning by repurposing a well-developed pre-trained model from a source domain to solve tasks in a target domain without model finetuning.
  arXiv  Detail & Related papers  (2022-02-22T02:33:54Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.