From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models

• URL: http://arxiv.org/abs/2511.02248v1
• Date: Tue, 04 Nov 2025 04:26:47 GMT
• Title: From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models
• Abstract summary: Existing solutions rely on static provisioning or model-level autoscaling.<n>This coarse-grained resource management leads to degraded performance or significant resource underutilization.<n>We propose an operator-level autoscaling framework, which allocates resources at finer (operator)-granularity.
• Score: 4.720658518775265
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Serving large generative models such as LLMs and multi- modal transformers requires balancing user-facing SLOs (e.g., time-to-first-token, time-between-tokens) with provider goals of efficiency and cost reduction. Existing solutions rely on static provisioning or model-level autoscaling, both of which treat the model as a monolith. This coarse-grained resource management leads to degraded performance or significant resource underutilization due to poor adaptability to dynamic inference traffic that is common online. The root cause of this inefficiency lies in the internal structure of generative models: they are executed as graphs of interconnected operators. Through detailed characterization and systematic analysis, we find that operators are heterogeneous in their compute and memory footprints and exhibit diverse sensitivity to workload and resource factors such as batch size, sequence length, and traffic rate. This heterogeneity suggests that the operator, rather than the entire model, is the right granularity for scaling decisions. We propose an operator-level autoscaling framework, which allocates resources at finer (operator)-granularity, optimizing the scaling, batching, and placement based on individual operator profiles. Evaluated on production-scale traces, our approach preserves SLOs with up to 40% fewer GPUs and 35% less energy, or under fixed resources achieves 1.6x higher throughput with 5% less energy. These results show that the operator, rather than the model, is fundamentally a more effective unit for scaling large generative workloads.

Related papers

• Multi-Agent Learning with Cooperation-Driven Optimization Dynamics [0.0]
  We propose a mechanism for cooperation among several artificial neural networks.<n>We consider several "small" agents, containing fewer parameters than a reference "large" one, that during training share their predictions.<n>We numerically compare the accuracy of those strategies on several standard benchmarks.
  arXiv  Detail & Related papers  (2026-09-15T09:51:54Z)
• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios [76.85739138203014]
  We present SpecFormer, a novel architecture that accelerates unidirectional and attention mechanisms.<n>We demonstrate that SpecFormer achieves lower training demands and reduced computational costs.
  arXiv  Detail & Related papers  (2025-11-25T14:20:08Z)
• ScaleDL: Towards Scalable and Efficient Runtime Prediction for Distributed Deep Learning Workloads [14.876533021201539]
  ScaleDL is a novel runtime prediction framework for deep neural networks (DNNs)<n>It combines nonlinear layer-wise modeling with graph neural network (GNN)-based cross-layer interaction mechanism.<n>Experiments show that ScaleDL enhances runtime prediction accuracy and generalizability, achieving 6 times lower MRE and 5 times lower RMSE compared to baseline models.
  arXiv  Detail & Related papers  (2025-11-06T08:05:55Z)
• The Curious Case of In-Training Compression of State Space Models [49.819321766705514]
  State Space Models (SSMs) tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference.<n>Key design challenge is striking the right balance between maximizing expressivity and limiting this computational burden.<n>Our approach, textscCompreSSM, applies to Linear Time-Invariant SSMs such as Linear Recurrent Units, but is also extendable to selective models.
  arXiv  Detail & Related papers  (2025-10-03T09:02:33Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Compute-Optimal Scaling for Value-Based Deep RL [99.680827753493]
  We investigate compute scaling for online, value-based deep RL.<n>Our analysis reveals a nuanced interplay between model size, batch size, and UTD.<n>We provide a mental model for understanding this phenomenon and build guidelines for choosing batch size and UTD.
  arXiv  Detail & Related papers  (2025-08-20T17:54:21Z)
• ACME: Adaptive Customization of Large Models via Distributed Systems [7.358399967930416]
  We propose ACME, an adaptive customization approach of Transformer-based large models via distributed systems.<n>ACME achieves cost-efficient models under model size constraints.<n>Average accuracy improves by 10 percent compared to the baseline.
  arXiv  Detail & Related papers  (2025-07-20T03:30:24Z)
• Deploying Large AI Models on Resource-Limited Devices with Split Federated Learning [39.73152182572741]
  This paper proposes a novel framework, named Quantized Split Federated Fine-Tuning Large AI Model (SFLAM)<n>By partitioning the training load between edge devices and servers, SFLAM can facilitate the operation of large models on devices.<n>SFLAM incorporates quantization management, power control, and bandwidth allocation strategies to enhance training efficiency.
  arXiv  Detail & Related papers  (2025-04-12T07:55:11Z)
• Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
  Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality.<n>We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
  arXiv  Detail & Related papers  (2024-11-30T10:56:30Z)
• CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction [77.8576094863446]
  We propose a new detextbfCoupled dutextbfAl-interactive lineatextbfR atttextbfEntion (CARE) mechanism. We first propose an asymmetrical feature decoupling strategy that asymmetrically decouples the learning process for local inductive bias and long-range dependencies. By adopting a decoupled learning way and fully exploiting complementarity across features, our method can achieve both high efficiency and accuracy.
  arXiv  Detail & Related papers  (2024-11-25T07:56:13Z)
• Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation [15.35494431928751]
  Transformer-based large language models (LLMs) exhibit impressive performance in generative tasks but also introduce significant challenges in real-world serving.<n>We introduce model-attention disaggregation to enhance the efficiency of LLM decoding.<n>We develop and deploy Lamina, an LLM inference system that incorporates model-attention disaggregation in a distributed heterogeneous cluster.
  arXiv  Detail & Related papers  (2024-05-03T02:15:15Z)
• Efficient generative adversarial networks using linear additive-attention Transformers [0.8287206589886879]
  We present a novel GAN architecture based on a linear attention Transformer block named Ladaformer.<n>LadaGAN consistently outperforms existing convolutional and Transformer GANs on benchmark datasets at different resolutions.<n>LadaGAN shows competitive performance compared to state-of-the-art multi-step generative models.
  arXiv  Detail & Related papers  (2024-01-17T21:08:41Z)
• A Generative Approach for Production-Aware Industrial Network Traffic Modeling [70.46446906513677]
  We investigate the network traffic data generated from a laser cutting machine deployed in a Trumpf factory in Germany. We analyze the traffic statistics, capture the dependencies between the internal states of the machine, and model the network traffic as a production state dependent process. We compare the performance of various generative models including variational autoencoder (VAE), conditional variational autoencoder (CVAE), and generative adversarial network (GAN)
  arXiv  Detail & Related papers  (2022-11-11T09:46:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.