Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

• URL: http://arxiv.org/abs/2607.20357v1
• Date: Wed, 22 Jul 2026 16:43:27 GMT
• Title: Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs
• Abstract summary: Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks.<n>Their high inference cost, arising from both the large number of input visual tokens and the heavy computation of the large language model (LLM), remains a key barrier to practical deployment.<n>We propose SmartVL, a unified adaptive inference framework that jointly controls vision token number and model compute capability.
• Score: 16.64366862436724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost, arising from both the large number of input visual tokens and the heavy computation of the large language model (LLM), remains a key barrier to practical deployment. Recent work attempts to reduce the cost by adaptively optimizing individual dimensions, e.g., pruning redundant visual tokens or skipping LLM layers and heads. Nonetheless, prior approaches typically treat these dimensions independently and overlook a fundamental coupling: the available compute resources must be dynamically allocated across all dimensions based on the input content. To bridge the gap, we propose SmartVL, a unified adaptive inference framework that jointly controls vision token number and model compute capability in response to varying input contents and compute budgets. SmartVL introduces a vision-side token controller that dynamically selects informative visual tokens and an LLM-side compute controller that adaptively adjusts LLM computation. Importantly, these controllers are trained to coordinate with each other so that the overall inference cost satisfies a target budget. To allow this joint scheduling, we connect the controllers using a shared budget encoding and leverage a differentiable latency estimator for end-to-end training. This design enables SmartVL to learn cross-stage allocation strategies that adapt to both input complexity and runtime compute constraints. Experiments across multiple MLLM benchmarks demonstrate that, with joint scheduling, SmartVL consistently outperforms prior adaptive methods and achieves superior accuracy-efficiency Pareto frontiers. Project page: https://www.schaterji.io/publications/2026/jointtokencompute.

Related papers

• CaRL-EM: Cost-Aware Reinforcement Learning for Entity Matching with LLMs [1.3673052804154124]
  CaRL-EM learns to dynamically plan the usage of inexpensive and expensive operators based on task complexity.<n>It consistently achieves a better quality-cost trade-off than strong LLM-based baselines.
  arXiv  Detail & Related papers  (2026-09-01T13:05:47Z)
• ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs [48.87461975674182]
  We introduce the Parallel Vision-Language (ParVL) scaling framework for Multimodal Large Language Models.<n>ParVL scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches.<n>We show that ParVL improves overall multimodal performance over same-recipe single-branch baselines.
  arXiv  Detail & Related papers  (2026-08-04T17:59:58Z)
• Compute Where it Counts: Self Optimizing Language Models [10.058821474955177]
  We study dynamic budget allocation for autoregressive decoding.<n>We train the policy with group-relative policy optimization on teacher-forced episodes.<n>Our reward trades off language-model quality against soft penalties that encourage episode-average budget usage.
  arXiv  Detail & Related papers  (2026-05-11T17:27:15Z)
• From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion [91.35078719566472]
  Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection.<n>We introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities.
  arXiv  Detail & Related papers  (2026-01-15T18:59:10Z)
• Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning [53.57360296655208]
  Large language models (LLMs) exhibit complementary strengths across domains and come with varying inference costs.<n>Existing approaches rely on decentralized frameworks, which invoke multiple LLMs for every input and thus lead to substantial and uncontrolled inference costs.<n>We introduce a centralized multi-LLM framework, where a controller LLM selectively coordinates a pool of expert models in a cost-efficient and cost-controllable manner.
  arXiv  Detail & Related papers  (2025-11-04T17:35:17Z)
• Learning to Inference Adaptively for Multimodal Large Language Models [19.510735093226703]
  We introduce AdaLLaVA, an adaptive inference framework that learns to reconfigure operations in an MLLM during inference.<n>Our results show that AdaLLaVA effectively adheres to input latency budget, achieving varying accuracy and latency tradeoffs at runtime.
  arXiv  Detail & Related papers  (2025-03-13T21:39:38Z)
• Cost-Optimal Grouped-Query Attention for Long-Context Modeling [45.981681856747365]
  Grouped-Query Attention (GQA) is a widely adopted strategy for reducing the computational cost of attention layers in large language models.<n>We analyze the relationship among context length, model size, GQA configuration, and model loss.<n>We propose a recipe for deriving cost-optimal GQA configurations.
  arXiv  Detail & Related papers  (2025-03-12T17:50:42Z)
• Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models [16.16372459671255]
  Large Language Models (LLMs) typically generate outputs token by token using a fixed compute budget. We propose a novel framework that integrates smaller auxiliary modules within each Feed-Forward Network layer of the LLM. We show that trained routers operate differently from oracles and often yield suboptimal solutions.
  arXiv  Detail & Related papers  (2024-10-01T16:10:21Z)
• CoMMIT: Coordinated Multimodal Instruction Tuning [90.1532838391285]
  Multimodal large language models (MLLMs) generally involve cooperative learning between a backbone LLM and a feature encoder of non-text input modalities.<n>In this paper, we analyze the MLLM instruction tuning from both theoretical and empirical perspectives.<n>We propose a Multimodal Balance Coefficient that enables quantitative measurement of the balance of learning.
  arXiv  Detail & Related papers  (2024-07-29T23:18:55Z)
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs for Embodied AI [10.82017289243097]
  Large Language Models (LLMs) are capable of reasoning over diverse input data modalities through pre-trained encoders. m-LLM improves the task accuracy by up to 4% compared to the best existing scheme.
  arXiv  Detail & Related papers  (2023-12-13T04:08:59Z)
• A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading [62.34538208323411]
  We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs) MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
  arXiv  Detail & Related papers  (2023-09-02T11:01:16Z)
• Efficient Controllable Multi-Task Architectures [85.76598445904374]
  We propose a multi-task model consisting of a shared encoder and task-specific decoders where both encoder and decoder channel widths are slimmable. Our key idea is to control the task importance by varying the capacities of task-specific decoders, while controlling the total computational cost. This improves overall accuracy by allowing a stronger encoder for a given budget, increases control over computational cost, and delivers high-quality slimmed sub-architectures.
  arXiv  Detail & Related papers  (2023-08-22T19:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.