Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling

• URL: http://arxiv.org/abs/2603.06403v1
• Date: Fri, 06 Mar 2026 15:45:33 GMT
• Title: Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling
• Abstract summary: We propose a multi-adapter-enhanced MLLM inference scheduling framework with three components.<n>emphM-CMAB consistently outperforms state-of-the-art baselines across budget regimes.
• Score: 19.731771957197044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal large language model (MLLM) inference scheduling enables strong response quality under practical and heterogeneous budgets, beyond what a homogeneous single-backend setting can offer. Yet online MLLM task scheduling is nontrivial, as requests vary sharply in modality composition and latent reasoning difficulty, while execution backends incur distinct, time-varying costs due to system jitter and network variation. These coupled uncertainties pose two core challenges: deriving semantically faithful yet scheduling-relevant multi-modal task representations, and making low-overhead online decisions over irreversible multi-dimensional budgets. Accordingly, we propose \emph{M-CMAB} (\underline{M}ulti-modal \underline{M}ulti-constraint \underline{C}ontextual \underline{M}ulti-\underline{A}rmed \underline{B}andit), a multi-adapter-enhanced MLLM inference scheduling framework with three components: (i) a CLS-attentive, frozen-backbone \emph{Predictor} that extracts compact task representations and updates only lightweight adapters for action-specific estimation; (ii) a primal-dual \emph{Constrainer} that maintains online Lagrange multipliers to enforce long-horizon constraints via per-round objectives; and (iii) a two-phase \emph{Scheduler} that balances exploration and exploitation under irreversible budgets. We establish a regret guarantee under multi-dimensional knapsack constraints. On a composite multimodal benchmark with heterogeneous backends, \emph{M-CMAB} consistently outperforms state-of-the-art baselines across budget regimes, achieving up to 14.18% higher reward and closely tracking an oracle-aided upper bound. Codes are available at https://anonymous.4open.science/r/M2CMAB/.

Related papers

• ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models [14.64726819144383]
  ZOMP is a query-efficient, fully forward-only prompt-tuning method.<n>It consistently outperforms prior BP-free prompt-tuning methods in both few-shot accuracy and query efficiency.<n>Our results show that jointly exploiting multimodality and low-rank structure is an effective route to practical, practical BP-free prompt tuning.
  arXiv  Detail & Related papers  (2026-08-08T10:55:58Z)
• WISERouter: LLM Routing with Workload Budget Constraint [18.88807929206437]
  Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale.<n>We introduce WISE (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration.<n>WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.
  arXiv  Detail & Related papers  (2026-07-26T17:20:39Z)
• PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization [49.659739762240655]
  Generative listwise ranking with Large Multimodal Models (LMMs) aims to capture global list context in a single forward pass, but its effectiveness degrades in long-context multimodal scenarios.<n>We identify a recurring failure mode, parse collapse, where the autoregressive decoder produces fluent yet incomplete rankings by silently omitting candidates and terminating early.<n>We propose PRISMR, a framework that replaces transient in-context list processing with parametric structural conditioning.
  arXiv  Detail & Related papers  (2026-06-11T06:09:51Z)
• MAny: Merge Anything for Multimodal Continual Instruction Tuning [52.50936513604062]
  textbfMAny (textbfMAny) is a framework that merges task-specific knowledge through textbfCross-modal textbfProjection textbfMerging.<n>textbfLow-rank textbfParameter textbfMerging (textbfLPM)
  arXiv  Detail & Related papers  (2026-04-15T15:57:23Z)
• Multinoulli Extension: A Lossless Continuous Relaxation for Partition-Constrained Subset Selection [60.07018090570548]
  We introduce a novel algorithm titled MultinoulliSCG, which is parameterfree and can achieve the same approximation guarantees as the distorted local-search methods.<n>We also present two novel online algorithms, namely, Multinoulli-CG and Multinoulli-GAGA, for the unexplored online subset selection problems over partition constraints.
  arXiv  Detail & Related papers  (2026-03-23T02:30:01Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• MagicAgent: Towards Generalized Agent Planning [73.21129030631421]
  We present textbfMagicAgent, a series of foundation models specifically designed for generalized agent planning.<n>We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks.<n>We show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T01:39:16Z)
• MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing [41.77627136743721]
  In practical deployments, workloads span lightweight OCR to complex multimodal reasoning.<n>routing is nontrivial due to modality fusion, wide variation in computational cost across models, and the absence of a standardized, budget-aware evaluation.<n>We present MMR-Bench, a unified benchmark that isolates the multimodal routing problem and enables comparison under fixed candidate sets and cost models.
  arXiv  Detail & Related papers  (2026-01-25T12:44:14Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Empowering Large Language Model for Sequential Recommendation via Multimodal Embeddings and Semantic IDs [28.752042722391934]
  Sequential recommendation (SR) aims to capture users' dynamic interests and sequential patterns based on their historical interactions.<n>MME-SID integrates multimodal embeddings and quantized embeddings to mitigate embedding collapse.<n>Extensive experiments on three public datasets validate the superior performance of MME-SID.
  arXiv  Detail & Related papers  (2025-09-02T07:02:29Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [57.577843653775]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for Chain-of-Thought Reasoning [60.84707424369494]
  Chain-of-Thought (CoT) reasoning has proven effective in enhancing large language models (LLMs) on complex tasks.<n>We introduce the Reasoning Boundary Framework++ (RBF++), a framework for evaluating and optimizing measurable boundaries of CoT capability.
  arXiv  Detail & Related papers  (2025-05-19T16:25:55Z)
• Cost-Effective Online Multi-LLM Selection with Versatile Reward Models [30.892090566736652]
  We introduce the textitC2MAB-V, an online model for selecting and using large language models (LLMs) textitC2MAB-V is specifically tailored for various collaborative task types with different reward models. We show that textitC2MAB-V effectively balances performance and cost-efficiency with nine LLMs for three application scenarios.
  arXiv  Detail & Related papers  (2024-05-26T14:38:24Z)
• Hierarchical Deep Reinforcement Learning Approach for Multi-Objective Scheduling With Varying Queue Sizes [19.457882940942284]
  We present MERLIN, a robust, modular and near-optimal DRL-based approach for multi-objective task scheduling. MERLIN applies a hierarchical approach to the MOTS problem by creating one neural network for the processing of individual tasks and another for the scheduling of the overall queue. In addition to being smaller and with shorted training times, the resulting architecture ensures that an item is processed in the same manner regardless of its position in the queue.
  arXiv  Detail & Related papers  (2020-07-17T21:59:06Z)
• Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints [52.58352707495122]
  We present a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination. We validate our results over a wide range of simulations on two distinct domains: multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
  arXiv  Detail & Related papers  (2020-05-27T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.