Token-Efficient Multimodal Reasoning via Image Prompt Packaging

• URL: http://arxiv.org/abs/2604.02492v1
• Date: Thu, 02 Apr 2026 19:50:59 GMT
• Title: Token-Efficient Multimodal Reasoning via Image Prompt Packaging
• Abstract summary: We introduce Image Prompt Packaging, a prompting paradigm that embeds structured text directly into images to reduce text token overhead.<n>We benchmark it across five datasets, three frontier models, and two task families.<n>We derive a cost decomposing savings by token type and show IPPg achieves 35.8--91.0% inference cost reductions.
• Score: 0.6465251961564605
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deploying large multimodal language models at scale is constrained by token-based inference costs, yet the cost-performance behavior of visual prompting strategies remains poorly characterized. We introduce Image Prompt Packaging (IPPg), a prompting paradigm that embeds structured text directly into images to reduce text token overhead, and benchmark it across five datasets, three frontier models (GPT-4.1, GPT-4o, Claude 3.5 Sonnet), and two task families (VQA and code generation). We derive a cost formulation decomposing savings by token type and show IPPg achieves 35.8--91.0\% inference cost reductions. Despite token compression of up to 96\%, accuracy remains competitive in many settings, though outcomes are highly model- and task-dependent: GPT-4.1 achieves simultaneous accuracy and cost gains on CoSQL, while Claude 3.5 incurs cost increases on several VQA benchmarks. Systematic error analysis yields a failure-mode taxonomy: spatial reasoning, non-English inputs, and character-sensitive operations are most vulnerable, while schema-structured tasks benefit most. A 125-configuration rendering ablation reveals accuracy shifts of 10--30 percentage points, establishing visual encoding choices as a first-class variable in multimodal system design.

Related papers

• Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration [71.77541419885925]
  We present a unified graph-based DP framework.<n>By modeling the dataset as a weighted graph, DP can be cast as a Maximum Weight Clique Problem (MWCP)<n>Our method outperforms existing DP methods while substantially reducing training cost, reducing training time by over 40% without sacrificing accuracy on ImageNet-1k with ResNet-50.
  arXiv  Detail & Related papers  (2026-06-11T05:13:32Z)
• Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data [6.3745740668603075]
  Large language models (LLMs) for domain-specific structured evaluation tasks incurs prohibitive latency, cost, and data-privacy overhead.<n>We present a hybrid framework that fine-tunes a small language model on only 219 curated examples and couples it with a deterministic rule-based postprocessing layer.<n>Our system achieves 100% structural validity, 83.0% human-validated overall accuracy, and 100% accuracy on the most critical classification field in blind evaluation on 53 unseen production transcripts.
  arXiv  Detail & Related papers  (2026-06-04T07:09:37Z)
• SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection [0.0]
  SpecKV is a lightweight adaptive controller that selects $$ per speculation step using signals extracted from the draft model itself.<n>We demonstrate that the optimal $$ shifts across compression regimes and that draft model confidence and entropy are strong predictors of acceptance rate.
  arXiv  Detail & Related papers  (2026-05-04T17:55:05Z)
• The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content [11.226669792850645]
  We formalise a phenomenon we term the structural attention tax: knowledge graph (KG) triples.<n>KG triples capture 2-3x more attention per token than semantically equivalent natural-language text.<n>We develop a formal framework decomposing attention scores into semantic and structural components.
  arXiv  Detail & Related papers  (2026-04-21T08:47:00Z)
• Towards Joint Quantization and Token Pruning of Vision-Language Models [53.978753457744055]
  Token pruning and low-bit quantization are complementary for reducing inference costs.<n>We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning.<n> Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime.
  arXiv  Detail & Related papers  (2026-04-19T08:18:29Z)
• VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents [1.06378109904813]
  VAREX is a benchmark for evaluating structured data extraction from government forms.<n>The benchmark comprises 1,777 documents with 1,771 unique documents with validated ground truth through three-phase quality assurance.<n>Results reveal that below 4B parameters, compliance output -- not extraction capability -- is a dominant bottleneck.
  arXiv  Detail & Related papers  (2026-03-16T11:15:56Z)
• Prism-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models [85.18197548789291]
  We propose PRISM-$$ (Projection-based Relevance-Informed Steering Method) to maximize discriminative energy while eliminating shared directions.<n> PRISM-$$ matches or exceeds the best existing method on 19 of 20 configurations, with relative gains up to +10.6%, while halving the cost of steering.<n> PRISM-$$ is compatible with FlashAttention and adds negligible memory overhead.
  arXiv  Detail & Related papers  (2026-03-11T12:24:45Z)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity [54.95089105944234]
  Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens.<n>We propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline.<n>We demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT.
  arXiv  Detail & Related papers  (2026-03-10T10:31:58Z)
• TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge [23.707347449137895]
  TernaryCLIP is a lightweight framework that converts connection weights of both vision and text encoders of CLIP into the ternary format.<n>Our work highlights the feasibility of extreme quantization for large multimodal models, supporting effective and efficient deployment on resource-constrained devices.
  arXiv  Detail & Related papers  (2025-10-23T14:53:32Z)
• Cost-Aware Contrastive Routing for LLMs [57.30288453580456]
  We introduce Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework that maps both prompts and models into a shared embedding space.<n>CSCR consistently outperforms baselines, improving the accuracy-cost tradeoff by up to 25%.
  arXiv  Detail & Related papers  (2025-08-17T20:16:44Z)
• FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Models [1.9894117371899613]
  Hallucinations in large language models pose a critical challenge for applications requiring factual reliability.<n>This work presents an effective approach for detecting and editing factually incorrect content in model-generated responses.
  arXiv  Detail & Related papers  (2025-07-28T15:41:53Z)
• UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks [60.46473247205654]
  Using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models.
  arXiv  Detail & Related papers  (2023-06-07T18:26:22Z)
• Expressive Losses for Verified Robustness via Convex Combinations [67.54357965665676]
  We study the relationship between the over-approximation coefficient and performance profiles across different expressive losses. We show that, while expressivity is essential, better approximations of the worst-case loss are not necessarily linked to superior robustness-accuracy trade-offs.
  arXiv  Detail & Related papers  (2023-05-23T12:20:29Z)
• Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens [65.4435926060951]
  We propose to significantly improve the efficiency of Transformers for ultra long sequences, by compressing the sequence into a much smaller representation at each layer. Our algorithm is not only efficient (achieving more than $3times$ efficiency gain compared to baselines on 4K and 16K lengths) but also offers competitive/better performance on a large number of tasks.
  arXiv  Detail & Related papers  (2023-05-07T10:32:18Z)
• Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition [2.6346614942667235]
  We study how to reduce the Conformer architecture complexity with a limited computing budget. We introduce progressive downsampling to the Conformer encoder and propose a novel attention mechanism named grouped attention. Within the same computing budget, the proposed architecture achieves better performances with faster training and decoding.
  arXiv  Detail & Related papers  (2021-08-31T07:48:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.