Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models

• URL: http://arxiv.org/abs/2506.12776v1
• Date: Sun, 15 Jun 2025 08:58:09 GMT
• Title: Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models
• Authors: Junbo Niu, Yuanhong Zheng, Ziyang Miao, Hejun Dong, Chunjiang Ge, Hao Liang, Ma Lu, Bohan Zeng, Qiahao Zheng, Conghui He, Wentao Zhang,
• Abstract summary: Vision-Language Models (VLMs) face significant challenges when dealing with the diverse resolutions and aspect ratios of real-world images.<n>We introduce RC-Bench, a novel benchmark designed to evaluate VLM capabilities under extreme visual conditions.<n>We also propose NativeRes-LLaVA, an open-source training framework that empowers VLMs to effectively process images at their native resolutions and aspect ratios.
• Score: 21.577488819845982
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) face significant challenges when dealing with the diverse resolutions and aspect ratios of real-world images, as most existing models rely on fixed, low-resolution inputs. While recent studies have explored integrating native resolution visual encoding to improve model performance, such efforts remain fragmented and lack a systematic framework within the open-source community. Moreover, existing benchmarks fall short in evaluating VLMs under varied visual conditions, often neglecting resolution as a critical factor. To address the "Resolution Dilemma" stemming from both model design and benchmark limitations, we introduce RC-Bench, a novel benchmark specifically designed to systematically evaluate VLM capabilities under extreme visual conditions, with an emphasis on resolution and aspect ratio variations. In conjunction, we propose NativeRes-LLaVA, an open-source training framework that empowers VLMs to effectively process images at their native resolutions and aspect ratios. Based on RC-Bench and NativeRes-LLaVA, we conduct comprehensive experiments on existing visual encoding strategies. The results show that Native Resolution Visual Encoding significantly improves the performance of VLMs on RC-Bench as well as other resolution-centric benchmarks. Code is available at https://github.com/Niujunbo2002/NativeRes-LLaVA.

Related papers

• Integrating Frequency-Domain Representations with Low-Rank Adaptation in Vision-Language Models [0.6715525121432597]
  This research presents a novel vision language model (VLM) framework to enhance feature extraction, scalability, and efficiency.<n>We evaluate the proposed model on caption generation and Visual Question Answering (VQA) tasks using benchmark datasets with varying levels of Gaussian noise.<n>Our model provides more detailed and contextually relevant responses, particularly for real-world images captured by a RealSense camera mounted on an Unmanned Ground Vehicle (UGV)
  arXiv  Detail & Related papers  (2025-03-08T01:22:10Z)
• REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models [67.55362046790512]
  Vision-language models lack the ability to correctly reason over spatial relationships. We develop the REVISION framework which improves spatial fidelity in vision-language models. Our results and findings indicate that utilizing rendering-based frameworks is an effective approach for developing spatially-aware models.
  arXiv  Detail & Related papers  (2024-08-05T04:51:46Z)
• INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model [71.50973774576431]
  We propose a novel MLLM, INF-LLaVA, designed for effective high-resolution image perception. We introduce a Dual-perspective Cropping Module (DCM), which ensures that each sub-image contains continuous details from a local perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to enable the mutual enhancement of global and local features.
  arXiv  Detail & Related papers  (2024-07-23T06:02:30Z)
• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models [81.71651422951074]
  Chain-of-Spot (CoS) method is a novel approach that enhances feature extraction by focusing on key regions of interest. This technique allows LVLMs to access more detailed visual information without altering the original image resolution. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content.
  arXiv  Detail & Related papers  (2024-03-19T17:59:52Z)
• MRET: Multi-resolution Transformer for Video Quality Assessment [37.355412115794195]
  No-reference video quality assessment (NR-VQA) for user generated content (UGC) is crucial for understanding and improving visual experience. Since large amounts of videos nowadays are 720p or above, the fixed and relatively small input used in conventional NR-VQA methods results in missing high-frequency details for many videos. We propose a novel Transformer-based NR-VQA framework that preserves the high-resolution quality information.
  arXiv  Detail & Related papers  (2023-03-13T21:48:49Z)
• Improving Performance of Object Detection using the Mechanisms of Visual Recognition in Humans [0.4297070083645048]
  We first track the performance of the state-of-the-art deep object recognition network, Faster- RCNN, as a function of image resolution. They also show that different spatial frequencies convey different information about the objects in recognition process. We propose a multi-resolution object recognition framework rather than a single-resolution network.
  arXiv  Detail & Related papers  (2023-01-23T19:09:36Z)
• ResFormer: Scaling ViTs with Multi-Resolution Training [100.01406895070693]
  We introduce ResFormer, a framework for improved performance on a wide spectrum of, mostly unseen, testing resolutions. In particular, ResFormer operates on replicated images of different resolutions and enforces a scale consistency loss to engage interactive information across different scales. We demonstrate, moreover, ResFormer is flexible and can be easily extended to semantic segmentation, object detection and video action recognition.
  arXiv  Detail & Related papers  (2022-12-01T18:57:20Z)
• Learning Resolution-Adaptive Representations for Cross-Resolution Person Re-Identification [49.57112924976762]
  Cross-resolution person re-identification problem aims to match low-resolution (LR) query identity images against high resolution (HR) gallery images. It is a challenging and practical problem since the query images often suffer from resolution degradation due to the different capturing conditions from real-world cameras. This paper explores an alternative SR-free paradigm to directly compare HR and LR images via a dynamic metric, which is adaptive to the resolution of a query image.
  arXiv  Detail & Related papers  (2022-07-09T03:49:51Z)
• Single Image Internal Distribution Measurement Using Non-Local Variational Autoencoder [11.985083962982909]
  This paper proposes a novel image-specific solution, namely non-local variational autoencoder (textttNLVAE) textttNLVAE is introduced as a self-supervised strategy that reconstructs high-resolution images using disentangled information from the non-local neighbourhood. Experimental results from seven benchmark datasets demonstrate the effectiveness of the textttNLVAE model.
  arXiv  Detail & Related papers  (2022-04-02T18:43:55Z)
• Cross-Resolution Adversarial Dual Network for Person Re-Identification and Beyond [59.149653740463435]
  Person re-identification (re-ID) aims at matching images of the same person across camera views. Due to varying distances between cameras and persons of interest, resolution mismatch can be expected. We propose a novel generative adversarial network to address cross-resolution person re-ID.
  arXiv  Detail & Related papers  (2020-02-19T07:21:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.