When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware

• URL: http://arxiv.org/abs/2608.03649v1
• Date: Tue, 04 Aug 2026 13:33:11 GMT
• Title: When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware
• Abstract summary: Fewer visual tokens do not guarantee lower end-to-end latency.<n>We evaluate break-even with a reproducible protocol that accounts for decision overhead, shared work, and the operators each policy can avoid.
• Score: 1.2103414068933553
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fewer visual tokens do not guarantee lower end-to-end latency. We evaluate break-even with a reproducible protocol that accounts for decision overhead, shared work, and the operators each policy can avoid. A stage-level decomposition reconciles these components with measured end-to-end latency. In a 30-example pilot, the two tested autoregressive probes remain slower than Full despite state reuse. A lightweight post-vision predictor yields paired confidence intervals below zero on RTX 3090 and A100 and remains significant after a conservative all-pairs Holm correction. A pre-vision image-size rule also yields intervals below zero on both GPUs, although neither comparison remains significant after the same correction. Pre-vision routing has a structural opportunity unavailable to post-vision pruning: it can avoid preprocessing and vision encoding. On A100, this opportunity outweighs a nearly eightfold larger downstream token reduction by the post-vision policy. Reported quality is conditional on examples answered correctly by Full and is not benchmark accuracy.

Related papers

• CFR without Unbiasedness: Deterministic Guarantees for Persistent Public-Chance Schedules [1.0390646006106319]
  We establish a deterministic target-transfer theorem for uniform, nonnested additive public cuts.<n>A component-resolved form of the theorem converts an execution trace into a numerical exploitability certificate.<n>A depth study locates a crossover between 32 and 64 full-cut outcome budgets, after which complete coverage dominates.
  arXiv  Detail & Related papers  (2026-08-14T09:28:35Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• Exactness Matters for Physical Rule Enforcement [4.230349220081444]
  It remains unclear when stronger physical rule enforcement becomes reliable and when it becomes a source of distribution shift.<n>We study this question through operator exactness, meaning whether the repair map is the identity on the target manifold.<n> Controlled mismatch, screened cleanup, adaptive gating, and external-backbone checks show that the best approximate-regime operating point can be raw or near-identity.
  arXiv  Detail & Related papers  (2026-05-08T07:59:59Z)
• Nondeterminism-Aware Optimistic Verification for Floating-Point Neural Networks [11.159101812934503]
  Nondeterministic Aware Optimistic verification protocol for floating-point execution on heterogeneous accelerators.<n>We implement NAO as a PyTorch-compatible runtime and a contract layer currently deployed on Holesky testnet.
  arXiv  Detail & Related papers  (2025-10-15T21:10:39Z)
• Distributed Training under Packet Loss [8.613477072763404]
  Leveraging unreliable connections will reduce latency but may sacrifice model accuracy and convergence once packets are dropped.<n>We introduce a principled, end-to-end solution that preserves accuracy and convergence guarantees under genuine packet loss.<n>This work bridges the gap between communication-efficient protocols and the accuracy and guarantees demanded by modern large-model training.
  arXiv  Detail & Related papers  (2025-07-02T11:07:20Z)
• One Sample is Enough to Make Conformal Prediction Robust [53.78604391939934]
  We show that conformal prediction attains some robustness even with a forward pass on a single randomly perturbed input.<n>Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. around 100) passes per input.
  arXiv  Detail & Related papers  (2025-06-19T19:14:25Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)
• On the Convergence of FedProx with Extrapolation and Inexact Prox [55.2480439325792]
  We investigate the behavior of FedExProx without the assumption that each client computes a certain operator exactly. We show that inexactness leads to convergence to a neighborhood of the solution. Our theoretical insights are validated through comprehensive numerical experiments.
  arXiv  Detail & Related papers  (2024-10-02T10:42:27Z)
• Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation [89.88214896713846]
  STA score considers two critical factors: temporal redundancy and semantic importance. We apply the STA module to off-the-shelf video Transformers and Videowins. Results: Kinetics-400 and Something-Something V2 achieve 30% overshelf reduction with a negligible 0.2% accuracy drop.
  arXiv  Detail & Related papers  (2023-08-08T19:38:15Z)
• Selective Network Linearization for Efficient Private Inference [49.937470642033155]
  We propose a gradient-based algorithm that selectively linearizes ReLUs while maintaining prediction accuracy. The results demonstrate up to $4.25%$ more accuracy (iso-ReLU count at 50K) or $2.2times$ less latency (iso-accuracy at 70%) than the current state of the art.
  arXiv  Detail & Related papers  (2022-02-04T19:00:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.