Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study

• URL: http://arxiv.org/abs/2604.25724v1
• Date: Tue, 28 Apr 2026 14:53:11 GMT
• Title: Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study
• Abstract summary: This paper presents a production deployment study of a modular, platform-agnostic inference architecture developed at Salesforce to support compound AI use cases.<n>Production results demonstrate over 50% reduction in tail latency (P95), up to 3.9x throughput improvement, and 30 to 40% cost savings compared to prior static deployments.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern enterprise AI applications increasingly rely on compound AI systems - architectures that compose multiple models, retrievers, and tools to accomplish complex tasks. Deploying such systems in production demands inference infrastructure that can efficiently serve concurrent, heterogeneous model invocations while maintaining cost-effectiveness and low latency. This paper presents a production deployment study of a modular, platform-agnostic inference architecture developed at Salesforce to support compound AI use cases including Agentforce (autonomous AI agents) and ApexGuru (AI-powered code analysis). The system integrates serverless execution, dynamic autoscaling, and MLOps pipelines to deliver consistent low-latency inference across multi-component agent workflows. We report production results demonstrating over 50% reduction in tail latency (P95), up to 3.9x throughput improvement, and 30 to 40% cost savings compared to prior static deployments. We further present a novel analysis of compound-system-specific challenges including multi-model fan-out overhead, cascading cold-start propagation, and heterogeneous scaling dynamics that emerge uniquely when serving agentic workloads. Through detailed case studies and operational lessons, we illustrate how the architecture enables compound AI systems to scale model invocations in parallel, handle bursty multi-agent workloads, and support rapid model iteration - capabilities essential for operationalizing agentic AI at enterprise scale.

Related papers

• BONSAI: A Mixed-Initiative Workspace for Human-AI Co-Development of Visual Analytics Applications [13.657176068149417]
  BONSAI is a mixed-initiative workspace for the multi-agent co-development of Visual Analytics applications.<n>This paper contributes a conceptual workflow, a scalable architecture, and an integrated system that successfully balances AI's generative speed with the structural rigor required for complex VA development.
  arXiv  Detail & Related papers  (2026-04-21T08:57:49Z)
• AIPC: Agent-Based Automation for AI Model Deployment with Qualcomm AI Runtime [0.0]
  This report presents AIPC, an AI agent-driven approach for constrained automation of AI model deployment.<n>It decomposes deployment into standardized, verifiable stages and injects deployment-domain knowledge into agent execution.<n>It can complete deployment from PyTorch to runnable QNN/SNPE inference within 7-20 minutes for structurally regular vision models.
  arXiv  Detail & Related papers  (2026-04-16T06:15:56Z)
• Multi-Agent Orchestration for High-Throughput Materials Screening on a Leadership-Class System [1.4669344809969722]
  Large language models (LLMs) play a critical role in autonomous decision-making.<n>We present a scalable, hierarchical multi-agent framework for orchestrating high- throughput screening campaigns.<n>Results demonstrate that the proposed agentic framework enables efficient and scalable execution on the Aurora supercomputer.
  arXiv  Detail & Related papers  (2026-04-09T01:01:11Z)
• AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery [49.77172027295716]
  AutoSOTA is an end-to-end automated research system that advances the latest State-Of-The-Art (SOTA) models published in top-tier AI papers.<n>It successfully discovers 105 new SOTA models that surpass the original reported methods, averaging approximately five hours per paper.
  arXiv  Detail & Related papers  (2026-04-07T07:52:01Z)
• An Empirical Study of Multi-Agent Collaboration for Automated Research [41.906658558789545]
  We present a study investigating the comparative efficacy of distinct multi-agent structures for automated machine learning optimization.<n>We benchmark a single-agent baseline against two multi-agent paradigms: a subagent architecture and an agent team architecture.<n>Our findings reveal a fundamental trade-off between operational stability and theoretical deliberation.
  arXiv  Detail & Related papers  (2026-03-31T11:57:00Z)
• ComAgent: Multi-LLM based Agentic AI Empowered Intelligent Wireless Networks [62.031889234230725]
  6G networks rely on complex cross-layer optimization.<n> manually translating high-level intents into mathematical formulations remains a bottleneck.<n>We present ComAgent, a multi-LLM agentic AI framework.
  arXiv  Detail & Related papers  (2026-01-27T13:43:59Z)
• MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era [74.42509044145417]
  MegaFlow is a large-scale distributed orchestration system that enables efficient scheduling, resource allocation, and fine-grained task management for agent-environment workloads.<n>In our agent training deployments, MegaFlow successfully orchestrates tens of thousands of concurrent agent tasks while maintaining high system stability and achieving efficient resource utilization.
  arXiv  Detail & Related papers  (2026-01-12T13:25:33Z)
• Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models [78.73992315826035]
  We introduce Youtu-LLM, a lightweight language model that harmonizes high computational efficiency with native agentic intelligence.<n>Youtu-LLM is pre-trained from scratch to systematically cultivate reasoning and planning capabilities.
  arXiv  Detail & Related papers  (2025-12-31T04:25:11Z)
• Towards Efficient Agents: A Co-Design of Inference Architecture and System [66.59916327634639]
  This paper presents AgentInfer, a unified framework for end-to-end agent acceleration.<n>We decompose the problem into four synergistic components: AgentCollab, AgentSched, AgentSAM, and AgentCompress.<n>Experiments on the BrowseComp-zh and DeepDiver benchmarks demonstrate that through the synergistic collaboration of these methods, AgentInfer reduces ineffective token consumption by over 50%.
  arXiv  Detail & Related papers  (2025-12-20T12:06:13Z)
• Efficient and Scalable Agentic AI with Heterogeneous Systems [1.8921715645847679]
  AI agents are emerging as a dominant workload in a wide range of applications, promising to be the vehicle that delivers the promised benefits of AI to enterprises and consumers.<n>To scale AI agent usage, we need efficient and scalable deployment and agent-serving infrastructure.<n>We present a system design for dynamic orchestration of AI agent workloads on heterogeneous compute infrastructure.
  arXiv  Detail & Related papers  (2025-07-25T19:02:42Z)
• ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems [80.69865295743149]
  This work attempts to study using LLM-based agents to design collaborative AI systems autonomously.<n>Based on ComfyBench, we develop ComfyAgent, a framework that empowers agents to autonomously design collaborative AI systems by generating.<n>While ComfyAgent achieves a comparable resolve rate to o1-preview and significantly surpasses other agents on ComfyBench, ComfyAgent has resolved only 15% of creative tasks.
  arXiv  Detail & Related papers  (2024-09-02T17:44:10Z)
• Inference Optimization of Foundation Models on AI Accelerators [68.24450520773688]
  Powerful foundation models, including large language models (LLMs), with Transformer architectures have ushered in a new era of Generative AI. As the number of model parameters reaches to hundreds of billions, their deployment incurs prohibitive inference costs and high latency in real-world scenarios. This tutorial offers a comprehensive discussion on complementary inference optimization techniques using AI accelerators.
  arXiv  Detail & Related papers  (2024-07-12T09:24:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.