Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

• URL: http://arxiv.org/abs/2608.07014v1
• Date: Fri, 07 Aug 2026 09:25:20 GMT
• Title: Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs
• Abstract summary: We show that Video LLMs improve smoothly or saturate as visual budgets grow.<n>We represent each frozen model--item pair by its response trajectory under controlled visual budgets.<n>We derive matched-grid measures of configuration complementarity, harmful transitions, and text overwrite.
• Score: 16.31176289941024
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Aggregate scaling curves suggest that Video LLMs improve smoothly or saturate as visual budgets grow. We show that this view can conceal large, opposing changes at the item level. We represent each frozen model--item pair by its response trajectory under controlled visual budgets and derive matched-grid measures of configuration complementarity, harmful transitions, and text overwrite. Across five open Video LLMs from three architecture families, four multiple-choice benchmark splits, open-ended QA and summarization, and fixed-history dialogue generation, no single budget serves all items. On the four-model matched MCQA grid, item-level oracle headroom spans $8.8$--$18.9$ accuracy points and $12.5$--$25.5\%$ of items are correct at a lower budget but wrong at a higher one. Task-appropriate continuous metrics show the same complementarity beyond multiple choice: Token-F1 oracle gaps are $2.7$--$3.7$ score points on MLVU generation and $3.8$--$4.8$ points on AVSD current-turn generation, even when mean quality improves with budget. The effect persists across frame count, spatial resolution, sampling policy, temporal--spatial allocation, and independently executed raw-video and cached pipelines, with per-item rates and membership tracking protocol choices. A controlled sampling intervention recovers $29.0\%$ of terminal regressions, and a structured frame audit identifies several recurring evidence pathways. We release per-item trajectories, protocol provenance, derived annotations, and reproducible analysis code as an auditing artifact. A confidence cascade matches fixed-$128f$ accuracy while reducing average shared frame cost by $31.7\%$, illustrating one operational use of the response matrix.

Related papers

• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding [17.716106078756066]
  Long-video understanding commonly compresses videos into a small set of frames or visual tokens for answer generation.<n>Our key idea is to organize selected frames into query-relevant cross-frame evidence before generation.<n>Across four benchmarks and two Video-MLLM backbones, GenEvA consistently improves matched-frame baselines.
  arXiv  Detail & Related papers  (2026-07-30T16:55:00Z)
• Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors [69.75895255395702]
  Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
  arXiv  Detail & Related papers  (2026-07-17T07:04:31Z)
• A Cost-Aware, Paired Protocol for Auditing Dynamic Tool Synthesis in Agentic Video Question Answering [2.1096366377985185]
  Agentic Video Questioning (VideoQA) systems invoke tools during inference, but their tool libraries are fixed.<n>We propose a cost-aware paired protocol for auditing tool-augmented video agents.
  arXiv  Detail & Related papers  (2026-07-01T21:01:57Z)
• FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&A [6.071121358322323]
  We introduce FLOWREADER, which reframes evidence assembly as a min-cost flow problem on a multimodal node graph.<n>The optimal flow is decomposed into candidate evidence paths, a compact non-redundant subset is selected by entropy-regularized replicator dynamics.<n>FLOWREADER is best on the two subsets dominated by fragmented evidence PaperTab and SlideVQA.
  arXiv  Detail & Related papers  (2026-06-05T13:05:00Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs [3.9266376632068485]
  GridProbe is an efficient training-free posterior-probing inference paradigm.<n>It scores evidence in answer space using a frozen VLM's own reasoning.<n>It selects question-relevant frames adaptively, resulting in sub-quadratic attention cost with little to no accuracy loss.
  arXiv  Detail & Related papers  (2026-05-11T15:57:46Z)
• CoV: Chain-of-View Prompting for Spatial Reasoning [64.77921266428824]
  Chain-of-View (CoV) prompting transforms a vision--language model into an active viewpoint reasoner.<n>We evaluate CoV on OpenEQA and obtain an average +11.56% improvement in LLM-Match, with a maximum gain of +13.62% on Qwen3-VL-Flash.
  arXiv  Detail & Related papers  (2026-01-08T17:59:42Z)
• A$^2$M$^2$-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition [56.79651392604733]
  A$2$M$2$-Net is able to handle the challenging temporal misalignment problem by establishing an adaptive alignment protocol for strong representation.<n>The experiments are conducted on five widely used FSAR benchmarks, and the results show our A$2$M$2$-Net achieves very competitive performance compared to state-of-the-arts.
  arXiv  Detail & Related papers  (2025-09-22T11:44:14Z)
• Flow-Guided Sparse Transformer for Video Deblurring [124.11022871999423]
  FlowGuided Sparse Transformer (F GST) is a framework for video deblurring. FGSW-MSA enjoys the guidance of the estimated optical flow to globally sample spatially sparse elements corresponding to the same scene patch in neighboring frames. Our proposed F GST outperforms state-of-the-art patches on both DVD and GOPRO datasets and even yields more visually pleasing results in real video deblurring.
  arXiv  Detail & Related papers  (2022-01-06T02:05:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.