OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Models

• URL: http://arxiv.org/abs/2511.00821v1
• Date: Sun, 02 Nov 2025 06:19:44 GMT
• Title: OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Models
• Abstract summary: Vision-Language Models (VLMs) have demonstrated strong performance across various tasks.<n>Position encoding plays a vital role in modeling both the sequential structure of textual information and the structure of visual information.<n>We propose OMEGA, a novel position encoding framework that employs Modality-Specific Position Scaling (MSPE) to assign positional indices.
• Score: 4.809606597614951
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have demonstrated strong performance across various multimodal tasks, where position encoding plays a vital role in modeling both the sequential structure of textual information and the spatial structure of visual information. However, current VLMs commonly adopt modality-unified 1D or 2D positional indexing strategies, which treat textual and visual tokens uniformly without accounting for their distinct structural properties and sequential continuity for text and spatial coherence for vision. To address this limitation, we propose OMEGA, a novel position encoding framework that employs Modality-Specific Position Encoding (MSPE) to assign positional indices while preserving the inherent structures of each modality across separate coordinate dimensions. Additionally, to align the information density of multimodal data in the positional index space, OMEGA introduces Global Adaptive Encoding Step Scaling (GAESS), which adaptively adjusts the position encoding step size of visual tokens based on the embedding entropy of both modalities. Experimental results demonstrate that OMEGA consistently enhances VLM performance across diverse architectures and VQA benchmarks. On visual-intensive tasks, OMEGA achieves up to 3.43% improvement over baseline position encoding strategies on Qwen2.5-VL-3B, with consistent gains observed across larger models including Qwen2.5-VL-7B and LLaVA-v1.5-7B.

Related papers

• UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval [87.12447641908143]
  We propose UniCVR, the first unified zero-shot composed visual retrieval framework.<n>UniCVR jointly addresses all three tasks without any task-specific human-annotated data.<n>In Stage I, we train the MLLM as a compositional query embedder via contrastive learning on a curated dataset of approximately 3.5M samples.<n>In Stage II, we introduce an MLLM-guided dual-level reranking mechanism that applies adaptive budgeted subset scoring to a small number of top-ranked candidates.
  arXiv  Detail & Related papers  (2026-04-22T08:16:50Z)
• MODIX: A Training-Free Multimodal Information-Driven Positional Index Scaling for Vision-Language Models [2.5982950439107064]
  Existing approaches uniformly assign positional indices to all tokens, overlooking variations in information density within and across modalities.<n>We propose MODIX, a training-free framework that dynamically adapts positional strides based on modality-specific contributions.<n>Experiments demonstrate that MODIX consistently improves multimodal reasoning and adaptively reallocates attention according to task-dependent information distributions.
  arXiv  Detail & Related papers  (2026-04-14T10:12:24Z)
• Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning [82.39668822222386]
  Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM)<n>We propose $textNwa$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity.<n>Experiments demonstrate that $textNwa$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%)
  arXiv  Detail & Related papers  (2026-02-03T00:51:03Z)
• 3DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scale [13.561331612635044]
  3DAlign-DAER is a framework designed to align text and 3D geometry via the proposed dynamic attention policy and the efficient retrieval strategy.<n>To facilitate text-3D alignment research and train our 3DAlign-DAER, we construct Align3D-2M, a large-scale dataset featuring 2M text-3D pairs.
  arXiv  Detail & Related papers  (2025-11-17T10:23:29Z)
• Learnable Spatial-Temporal Positional Encoding for Link Prediction [44.0907827498725]
  We propose a simple temporal link prediction model named L-STEP.<n>L-STEP can preserve the graph property from the spatial-temporal spectral viewpoint.<n>L-STEP obtains the leading performance in the newest large-scale TGB benchmark.
  arXiv  Detail & Related papers  (2025-06-10T00:35:53Z)
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models [68.46716645478661]
  Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content.<n>Current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints.<n>We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation.
  arXiv  Detail & Related papers  (2025-05-27T17:59:26Z)
• econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians [56.85804719947]
  We propose econSG for open-vocabulary semantic segmentation with 3DGS.<n>Our econSG shows state-of-the-art performance on four benchmark datasets compared to the existing methods.
  arXiv  Detail & Related papers  (2025-04-08T13:12:31Z)
• Tabular Embeddings for Tables with Bi-Dimensional Hierarchical Metadata and Nesting [0.7421845364041001]
  We introduce novel specialized embeddings optimized, and explicitly tailored to encode the intricacies of complex 2-D context in tables.<n>Our solution outperforms the state-of-the-art models with the significant MAP delta of up to 0.28.
  arXiv  Detail & Related papers  (2025-02-20T01:04:11Z)
• Exploring Primitive Visual Measurement Understanding and the Role of Output Format in Learning in Vision-Language Models [46.28245341706191]
  This work investigates the capabilities of vision quadrant models (VLMs) in visual understanding and measurement of primitive shapes.<n>We fine-tune state-of-the-art VLMs using Low-Rank Adaptation (LoRA) and validate them on multiple out-of-domain (OD) scenarios.
  arXiv  Detail & Related papers  (2025-01-25T09:01:37Z)
• Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks [41.488394198111976]
  Vision language models (VLMs) like CLIP show stellar zero-shot capability on classification benchmarks.<n> selecting the VLM with the highest performance on the unlabeled downstream task is non-trivial.<n>This paper introduces the problem of textbfunsupervised vision-language model selection, where only unsupervised downstream datasets are available.
  arXiv  Detail & Related papers  (2024-12-30T03:26:53Z)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [57.92316645992816]
  Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space.<n>We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs)<n>We demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.
  arXiv  Detail & Related papers  (2024-12-03T06:15:04Z)
• Aligning Large Language Models and Geometric Deep Models for Protein Representation [57.59506688299817]
  Latent representation alignment is used to map embeddings from different modalities into a shared space, often aligned with the embedding space of large language models (LLMs)<n>Preliminary protein-focused large language models (MLLMs) have emerged, but they have predominantly relied on approaches lacking a fundamental understanding of optimal alignment practices across representations.<n>In this study, we explore the alignment of multimodal representations between LLMs and Geometric Deep Models (GDMs) in the protein domain.<n>Our work examines alignment factors from both model and protein perspectives, identifying challenges in current alignment methodologies and proposing strategies to improve the alignment process.
  arXiv  Detail & Related papers  (2024-11-08T04:15:08Z)
• Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment [130.15775113897553]
  Finsta is a fine-grained structural-temporal alignment learning method. It consistently improves the existing 13 strong-tuning video-language models.
  arXiv  Detail & Related papers  (2024-06-27T15:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.