Exploring Primitive Visual Measurement Understanding and the Role of Output Format in Learning in Vision-Language Models

• URL: http://arxiv.org/abs/2501.15144v2
• Date: Tue, 02 Sep 2025 09:32:21 GMT
• Title: Exploring Primitive Visual Measurement Understanding and the Role of Output Format in Learning in Vision-Language Models
• Abstract summary: This work investigates the capabilities of vision quadrant models (VLMs) in visual understanding and measurement of primitive shapes.<n>We fine-tune state-of-the-art VLMs using Low-Rank Adaptation (LoRA) and validate them on multiple out-of-domain (OD) scenarios.
• Score: 46.28245341706191
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This work investigates the capabilities of current vision-language models (VLMs) in visual understanding and attribute measurement of primitive shapes using a benchmark focused on controlled 2D shape configurations with variations in spatial positioning, occlusion, rotation, size, and shape attributes such as type, quadrant, center-coordinates, rotation, occlusion status, and color as shown in Figure 1 and supplementary Figures S3-S81. We fine-tune state-of-the-art VLMs (2B-8B parameters) using Low-Rank Adaptation (LoRA) and validate them on multiple out-of-domain (OD) scenarios from our proposed benchmark. Our findings reveal that coherent sentence-based outputs outperform tuple formats, particularly in OD scenarios with large domain gaps. Additionally, we demonstrate that scaling numeric tokens during loss computation enhances numerical approximation capabilities, further improving performance on spatial and measurement tasks. These results highlight the importance of output format design, loss scaling strategies, and robust generalization techniques in enhancing the training and fine-tuning of VLMs, particularly for tasks requiring precise spatial approximations and strong OD generalization.

Related papers

• PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought [19.680475189691965]
  Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures.<n>We propose PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM)<n>We design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures.
  arXiv  Detail & Related papers  (2026-06-23T13:06:51Z)
• Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds [53.82500407523346]
  PointINS is an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning.<n>PointINS achieves on average +3.5% mAP improvement for indoor instance segmentation and +4.1% PQ gain for outdoor panoptic segmentation.
  arXiv  Detail & Related papers  (2026-03-26T08:31:06Z)
• MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing [117.58619053719251]
  MinerU2.5 is a document parsing model that achieves state-of-the-art recognition accuracy while maintaining exceptional computational efficiency.<n>Our approach employs a coarse-to-fine, two-stage parsing strategy that decouples global layout analysis from local content recognition.
  arXiv  Detail & Related papers  (2025-09-26T10:45:48Z)
• RangeSAM: Leveraging Visual Foundation Models for Range-View repesented LiDAR segmentation [6.513648249086729]
  We present the first range-view framework that adapts SAM2 to 3D segmentation, coupling efficient 2D feature extraction with standard projection/back-projection to operate on point clouds.<n>Our approach achieves competitive performance on Semantic KITTI while benefiting from the speed, scalability, and deployment simplicity of 2D-centric pipelines.
  arXiv  Detail & Related papers  (2025-09-19T11:33:10Z)
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception [71.26728044621458]
  DeCLIP is a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively.<n>It consistently achieves state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
  arXiv  Detail & Related papers  (2025-08-15T06:43:51Z)
• Hi^2-GSLoc: Dual-Hierarchical Gaussian-Specific Visual Relocalization for Remote Sensing [6.997091164331322]
  Visual relocalization is fundamental to remote sensing and UAV applications.<n>Existing methods face inherent trade-offs: image-based retrieval and pose regression approaches lack precision.<n>We introduce $mathrmHi2$-GSLoc, a dual-hierarchical relocalization framework that follows a sparse-to-dense and coarse-to-fine paradigm.
  arXiv  Detail & Related papers  (2025-07-21T14:47:56Z)
• Intern-GS: Vision Model Guided Sparse-View 3D Gaussian Splatting [95.61137026932062]
  Intern-GS is a novel approach to enhance the process of sparse-view Gaussian splatting.<n>We show that Intern-GS achieves state-of-the-art rendering quality across diverse datasets.
  arXiv  Detail & Related papers  (2025-05-27T05:17:49Z)
• Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model [0.8747606955991707]
  We propose a vision-language model that incorporates architectural improvements to enhance segmentation accuracy and cross-modal alignment.<n>SegVLM shows strong generalization across diverse datasets and referring expression scenarios.
  arXiv  Detail & Related papers  (2025-05-25T17:42:53Z)
• Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models [85.51753014478315]
  We introduce AdaptPrune, a novel plug-and-play training-free pruning method. It builds on conventional attention-based pruning by integrating spatial distance and token similarity with an adaptive NMS approach. Our approach ensures a comprehensive evaluation of token importance and substantially refines the pruning decisions.
  arXiv  Detail & Related papers  (2025-03-11T03:58:17Z)
• Integrating Frequency-Domain Representations with Low-Rank Adaptation in Vision-Language Models [0.6715525121432597]
  This research presents a novel vision language model (VLM) framework to enhance feature extraction, scalability, and efficiency. We evaluate the proposed model on caption generation and Visual Question Answering (VQA) tasks using benchmark datasets with varying levels of Gaussian noise. Our model provides more detailed and contextually relevant responses, particularly for real-world images captured by a RealSense camera mounted on an Unmanned Ground Vehicle (UGV)
  arXiv  Detail & Related papers  (2025-03-08T01:22:10Z)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [57.92316645992816]
  Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space. We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs) We demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.
  arXiv  Detail & Related papers  (2024-12-03T06:15:04Z)
• HRVMamba: High-Resolution Visual State Space Model for Dense Prediction [60.80423207808076]
  State Space Models (SSMs) with efficient hardware-aware designs have demonstrated significant potential in computer vision tasks. These models have been constrained by three key challenges: insufficient inductive bias, long-range forgetting, and low-resolution output representation. We introduce the Dynamic Visual State Space (DVSS) block, which employs deformable convolution to mitigate the long-range forgetting problem. We also introduce High-Resolution Visual State Space Model (HRVMamba) based on the DVSS block, which preserves high-resolution representations throughout the entire process.
  arXiv  Detail & Related papers  (2024-10-04T06:19:29Z)
• REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models [67.55362046790512]
  Vision-language models lack the ability to correctly reason over spatial relationships. We develop the REVISION framework which improves spatial fidelity in vision-language models. Our results and findings indicate that utilizing rendering-based frameworks is an effective approach for developing spatially-aware models.
  arXiv  Detail & Related papers  (2024-08-05T04:51:46Z)
• Efficient Visual State Space Model for Image Deblurring [83.57239834238035]
  Convolutional neural networks (CNNs) and Vision Transformers (ViTs) have achieved excellent performance in image restoration. We propose a simple yet effective visual state space model (EVSSM) for image deblurring.
  arXiv  Detail & Related papers  (2024-05-23T09:13:36Z)
• Condition-Invariant Semantic Segmentation [77.10045325743644]
  We implement Condition-Invariant Semantic (CISS) on the current state-of-the-art domain adaptation architecture.<n>Our method achieves the second-best performance on the normal-to-adverse Cityscapes$to$ACDC benchmark.<n>CISS is shown to generalize well to domains unseen during training, such as BDD100K-night and ACDC-night.
  arXiv  Detail & Related papers  (2023-05-27T03:05:07Z)
• ReF -- Rotation Equivariant Features for Local Feature Matching [30.459559206664427]
  We propose an alternative, complementary approach that centers on inducing bias in the model architecture itself to generate rotation-specific' features. We demonstrate that this high performance, rotation-specific coverage from the steerable CNNs can be expanded to all rotation angles. We present a detailed analysis of the performance effects of ensembling, robust estimation, network architecture variations, and the use of rotation priors.
  arXiv  Detail & Related papers  (2022-03-10T07:36:09Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)
• Self-supervised Geometric Perception [96.89966337518854]
  Self-supervised geometric perception is a framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels. We show that SGP achieves state-of-the-art performance that is on-par or superior to the supervised oracles trained using ground-truth labels.
  arXiv  Detail & Related papers  (2021-03-04T15:34:43Z)
• Consistent Representation Learning for High Dimensional Data Analysis [30.122549443821974]
  High dimensional data analysis includes three fundamental tasks: dimensionality reduction, clustering, and visualization. Inconsistencies can occur when the three associated tasks are done separately. We propose a novel neural network-based method, called Consistent Representation Learning, to accomplish the three associated tasks end-to-end.
  arXiv  Detail & Related papers  (2020-12-01T13:39:50Z)
• Improving the generalization of network based relative pose regression: dimension reduction as a regularizer [16.63174637692875]
  State-of-the-art visual localization methods perform pose estimation using geometry based solver within the RANSAC framework. End-to-end learning based regression networks provide a solution to circumvent the requirement for precise pixel-level correspondences. In this paper, we explicitly add a learnable matching layer within the network to isolate the pose regression solver from the absolute image feature values. We implement this dimension regularization strategy within a two-layer pyramid based framework to regress the localization results from coarse to fine.
  arXiv  Detail & Related papers  (2020-10-24T06:20:46Z)
• Spatial-Scale Aligned Network for Fine-Grained Recognition [42.71878867504503]
  Existing approaches for fine-grained visual recognition focus on learning marginal region-based representations. We propose the spatial-scale aligned network (SSANET) and implicitly address misalignments during the recognition process.
  arXiv  Detail & Related papers  (2020-01-05T11:12:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.