Adaptive Vision-Language Model Routing for Computer Use Agents

• URL: http://arxiv.org/abs/2603.12823v1
• Date: Fri, 13 Mar 2026 09:21:25 GMT
• Title: Adaptive Vision-Language Model Routing for Computer Use Agents
• Abstract summary: Computer Use Agents translate instructions into actions such as clicks, keystrokes, and scrolls.<n>Current CUA systems typically route every action to a single fixed model regardless of difficulty.<n>We propose Confusedbf VLM Routing (AVR), a framework that inserts a lightweight semantic routing layer between the CUA orchestrator and a pool of VLMs.
• Score: 9.457255218406333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computer Use Agents (CUAs) translate natural-language instructions into Graphical User Interface (GUI) actions such as clicks, keystrokes, and scrolls by relying on a Vision-Language Model (VLM) to interpret screenshots and predict grounded tool calls. However, grounding accuracy varies dramatically across VLMs, while current CUA systems typically route every action to a single fixed model regardless of difficulty. We propose \textbf{Adaptive VLM Routing} (AVR), a framework that inserts a lightweight semantic routing layer between the CUA orchestrator and a pool of VLMs. For each tool call, AVR estimates action difficulty from multimodal embeddings, probes a small VLM to measure confidence, and routes the action to the cheapest model whose predicted accuracy satisfies a target reliability threshold. For \textit{warm} agents with memory of prior UI interactions, retrieved context further narrows the capability gap between small and large models, allowing many actions to be handled without escalation. We formalize routing as a cost--accuracy trade-off, derive a threshold-based policy for model selection, and evaluate AVR using ScreenSpot-Pro grounding data together with the OpenClaw agent routing benchmark. Across these settings, AVR projects inference cost reductions of up to 78\% while staying within 2 percentage points of an all-large-model baseline. When combined with the Visual Confused Deputy guardrail, AVR also escalates high-risk actions directly to the strongest available model, unifying efficiency and safety within a single routing framework. Materials are also provided Model, benchmark, and code: https://github.com/vllm-project/semantic-router.

Related papers

• Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference [5.997400745120167]
  We propose a token-aware progressive model routing method with adaptive edge-cloud collaboration for efficient multimodal inference.<n>Pro-Theater employs a two-stage progressive decision mechanism. First, a lightweight prompt pre-scorer module performs rapid pre-screening before token generation begins.<n>Second, a token-aware verifier reads the sampling probability distribution of each token the small model generates, estimating the model's confidence in its own output to determine, per request, whether the answer ships or escalates.
  arXiv  Detail & Related papers  (2026-08-28T16:45:58Z)
• SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks [11.50404356970464]
  Large language models (LLMs) embedded in multi-turn agentic harnesses are reshaping software engineering (SWE)<n>But routing every task to a frontier model is wasteful when many issues admit cheap fixes.<n>We introduce SWE--a value-based temporal approach that lets a cheap model run for a few exploratory turns and reads the resulting partial trajectory.<n>We show that SWE- greatly improves the cost efficiency of SWE tasks, while maintaining the majority of the performances of the stronger model.
  arXiv  Detail & Related papers  (2026-06-30T01:46:26Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents [18.115492558482995]
  We introduce WildRoadBench, a wild aerial road-damage grounding benchmark.<n>It couples direct visual grounding by vision-feedback models with autonomous research-and-engineering by LLM-driven agents.<n>We benchmark a broad pool of closed-source frontier models and open-source VLMs together with several frontier LLM-driven agents.
  arXiv  Detail & Related papers  (2026-05-19T15:08:34Z)
• LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer? [69.71754384259167]
  Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency.<n>We propose Latent, a router that formulates MLLM routing as counterfactual multimodal utility prediction.<n> Experiments on MMR-Bench and VL-Bench show that Latent outperforms fixed-model, feature-level, and learned-router baselines.
  arXiv  Detail & Related papers  (2026-05-11T22:42:12Z)
• Step-level Optimization for Efficient Computer-use Agents [51.29573359027217]
  We argue that strong computer-use agents remain expensive and slow in practice.<n>We propose an event-driven, step-level cascade for computer-use agents.
  arXiv  Detail & Related papers  (2026-04-29T19:59:36Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents [59.107657859025586]
  GUI-360$circ$ is a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs)<n>The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications.<n>The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space.
  arXiv  Detail & Related papers  (2025-11-06T12:19:02Z)
• ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Models [26.059355108708374]
  We propose ECVL-ROUTER, the first scenario-aware routing framework for Vision-Language Models (VLMs)<n>Our approach introduces a new routing strategy and evaluation metrics that dynamically select the appropriate model for each query based on user requirements.<n>Results show that our approach successfully routes over 80% of queries to the small model while incurring less than 10% drop in problem solving probability.
  arXiv  Detail & Related papers  (2025-10-31T07:46:44Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• On the Practice of Deep Hierarchical Ensemble Network for Ad Conversion Rate Prediction [14.649184507551436]
  We propose a multitask learning framework with DHEN as the single backbone model architecture to predict all CVR tasks.<n>We build both on-site real-time user behavior sequences and off-site conversion event sequences for CVR prediction purposes.<n>Our method achieves state-of-the-art performance compared to previous single feature crossing modules with pre-trained user personalization features.
  arXiv  Detail & Related papers  (2025-04-10T23:41:34Z)
• Vision-Language Models Create Cross-Modal Task Representations [58.19152818504624]
  We find that vision-language models (VLMs) can align conceptually equivalent inputs into a shared task vector.<n>We measure this alignment via cross-modal transfer on a range of tasks and model architectures.<n>We show that task vectors can be transferred from a base language model to its fine-tuned vision-language counterpart.
  arXiv  Detail & Related papers  (2024-10-29T17:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.