Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models

• URL: http://arxiv.org/abs/2512.15372v1
• Date: Wed, 17 Dec 2025 12:19:54 GMT
• Title: Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models
• Authors: Mikel Williams-Lekuona, Georgina Cosma,
• Abstract summary: Vision transformers in vision-language models apply uniform computational effort across all images, expending 175.33 GFLOPs (ViT-L/14)<n>We propose ICAR (Image Complexity-Aware Retrieval), which enables vision transformers to use less compute for simple images.
• Score: 0.17188280334580197
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision transformers in vision-language models apply uniform computational effort across all images, expending 175.33 GFLOPs (ViT-L/14) whether analysing a straightforward product photograph or a complex street scene. We propose ICAR (Image Complexity-Aware Retrieval), which enables vision transformers to use less compute for simple images whilst processing complex images through their full network depth. The key challenge is maintaining cross-modal alignment: embeddings from different processing depths must remain compatible for text matching. ICAR solves this through dual-path training that produces compatible embeddings from both reduced-compute and full-compute processing. This maintains compatibility between image representations and text embeddings in the same semantic space, whether an image exits early or processes fully. Unlike existing two-stage approaches that require expensive reranking, ICAR enables direct image-text matching without additional overhead. To determine how much compute to use, we develop ConvNeXt-IC, which treats image complexity assessment as a classification task. By applying modern classifier backbones rather than specialised architectures, ConvNeXt-IC achieves state-of-the-art performance with 0.959 correlation with human judgement (Pearson) and 4.4x speedup. Evaluated on standard benchmarks augmented with real-world web data, ICAR achieves 20% practical speedup while maintaining category-level performance and 95% of instance-level performance, enabling sustainable scaling of vision-language systems.

Related papers

• DReX: Pure Vision Fusion of Self-Supervised and Convolutional Representations for Image Complexity Prediction [1.771934382051849]
  We propose a vision-only model that fuses self-supervised and convolutional representations to predict image complexity.<n>DReX achieves state-of-the-art performance on the IC9600 benchmark.<n>Our findings suggest that visual features alone can be sufficient for human-aligned complexity prediction.
  arXiv  Detail & Related papers  (2025-11-21T06:57:33Z)
• Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents [99.62178668680578]
  We propose Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer.<n> VC2L operates entirely in pixel space by rendering all inputs, whether textual, visual, or combined, as images.<n>To capture complex cross-modal relationships in web documents, VC2L employs a snippet-level contrastive learning objective that aligns consecutive multimodal segments.
  arXiv  Detail & Related papers  (2025-10-21T14:59:29Z)
• ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution [71.69364653858447]
  Existing Multimodal Large Language Models (MLLMs) suffer from increased inference costs due to the additional vision tokens introduced by image inputs.<n>We propose Visual Consistency Learning (ViCO), a novel training algorithm that enables the model to represent images of varying complexities using different numbers of vision tokens.<n> Experimental results demonstrate that our method can reduce the number of vision tokens by up to 50% while maintaining the model's perception, reasoning, and OCR capabilities.
  arXiv  Detail & Related papers  (2025-10-14T17:58:10Z)
• Text to Image for Multi-Label Image Recognition with Joint Prompt-Adapter Learning [69.33115351856785]
  We present a novel method, called T2I-PAL, to tackle the modality gap issue when using only text captions for PEFT.<n>The core design of T2I-PAL is to leverage pre-trained text-to-image generation models to generate photo-realistic and diverse images from text captions.<n>Extensive experiments on multiple benchmarks, including MS-COCO, VOC2007, and NUS-WIDE, show that our T2I-PAL can boost recognition performance by 3.47% in average.
  arXiv  Detail & Related papers  (2025-06-12T11:09:49Z)
• Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency [41.87857129429512]
  We introduce the Adventurer series models where we treat images as sequences of patch tokens and employ uni-directional language models to learn visual representations.<n>This modeling paradigm allows us to process images in a recurrent formulation with linear complexity relative to the sequence length.<n>In detail, we introduce two simple designs that seamlessly integrate image inputs into the causal inference framework.
  arXiv  Detail & Related papers  (2024-10-10T04:14:52Z)
• Optimizing CLIP Models for Image Retrieval with Maintained Joint-Embedding Alignment [0.7499722271664144]
  Contrastive Language and Image Pairing (CLIP) is a transformative method in multimedia retrieval. CLIP typically trains two neural networks concurrently to generate joint embeddings for text and image pairs. This paper addresses the challenge of optimizing CLIP models for various image-based similarity search scenarios.
  arXiv  Detail & Related papers  (2024-09-03T14:33:01Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [52.598772767324036]
  We introduce Image-GS, a content-adaptive image representation based on 2D Gaussians radiance.<n>It supports hardware-friendly rapid access for real-time usage, requiring only 0.3K MACs to decode a pixel.<n>We demonstrate its versatility with several applications, including texture compression, semantics-aware compression, and joint image compression and restoration.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval [92.13664084464514]
  The task of composed image retrieval (CIR) aims to retrieve images based on the query image and the text describing the users' intent. Existing methods have made great progress with the advanced large vision-language (VL) model in CIR task, however, they generally suffer from two main issues: lack of labeled triplets for model training and difficulty of deployment on resource-restricted environments. We propose Image2Sentence based Asymmetric zero-shot composed image retrieval (ISA), which takes advantage of the VL model and only relies on unlabeled images for composition learning.
  arXiv  Detail & Related papers  (2024-03-03T07:58:03Z)
• COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval [59.15034487974549]
  We propose a novel COllaborative Two-Stream vision-language pretraining model termed COTS for image-text retrieval. Our COTS achieves the highest performance among all two-stream methods and comparable performance with 10,800X faster in inference. Importantly, our COTS is also applicable to text-to-video retrieval, yielding new state-ofthe-art on the widely-used MSR-VTT dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:34:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.