Mitigating Coordinate Prediction Bias from Positional Encoding Failures

• URL: http://arxiv.org/abs/2510.22102v1
• Date: Sat, 25 Oct 2025 00:58:47 GMT
• Title: Mitigating Coordinate Prediction Bias from Positional Encoding Failures
• Abstract summary: High-resolution inputs exacerbate this difficulty by producing long token sequences that weaken positional encodings.<n>We analyze how MLLMs behave when visual positional encodings are deliberately perturbed through shuffling.<n>We propose Vision-PE Shuffle Guidance (VPSG), a training-free test-time method that leverages the directional nature of these biases for correction.
• Score: 31.63900089866925
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) excel at vision-language tasks such as VQA and document understanding, yet precise coordinate prediction remains challenging. High-resolution inputs exacerbate this difficulty by producing long token sequences that weaken positional encodings and introduce directional biases in coordinate outputs. We investigate this phenomenon by analyzing how MLLMs behave when visual positional encodings (VPEs) are deliberately perturbed through shuffling. Our analysis reveals that such perturbations induce predictable, non-random coordinate biases rather than random errors, suggesting that models rely on internal positional priors when spatial grounding signals are degraded. Crucially, we observe similar directional error patterns in natural high-resolution datasets, indicating that positional encoding failures are a key bottleneck for accurate coordinate prediction at scale. To address this issue, we propose Vision-PE Shuffle Guidance (VPSG), a training-free test-time method that leverages the directional nature of these biases for correction. VPSG runs auxiliary decoding with shuffled VPEs to isolate position-unconditioned tendencies, then uses this as negative evidence to guide digit prediction while preserving coordinate format through a lightweight finite-state machine. Experiments on ScreenSpot-Pro demonstrate reliable improvements, highlighting positional encoding robustness as a critical factor for spatial reasoning in MLLMs.

Related papers

• DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers [4.777114198703681]
  Camera-based positional encoding is crucial for providing spatial cues in 3D computer vision.<n>In this paper, we propose a novel camera-based positional encoding that explicitly decouples rotation and translation.<n>Extensive evaluations on NVS tasks demonstrate that DPPE enables stable long-term training even in scaled-up training setup.
  arXiv  Detail & Related papers  (2026-06-30T12:38:58Z)
• Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration [36.91587844807066]
  Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval.<n>Yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance.<n>We propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction.
  arXiv  Detail & Related papers  (2026-05-12T06:17:01Z)
• Relaxing Positional Alignment in Masked Diffusion Language Models [6.511565218210195]
  Masked diffusion language models (MDLMs) have emerged as a promising alternative to dominant autoregressive approaches.<n>We show that strict positional prediction makes MDLM decoding highly sensitive to token misalignment.<n>We apply this approach to the widely used MDLM model and conduct experiments on five open-ended text generation benchmarks.
  arXiv  Detail & Related papers  (2026-01-30T13:09:21Z)
• SMKC: Sketch Based Kernel Correlation Images for Variable Cardinality Time Series Anomaly Detection [0.0]
  In operational environments, monitoring systems frequently experience sensor churn.<n>We propose SMKC, a framework that decouples the dynamic input structure from the anomaly detector.<n>We find that a detector using random projections and nearest neighbors on the SMKC representation performs competitively with fully trained baselines.
  arXiv  Detail & Related papers  (2026-01-28T21:15:11Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• Adapting Language Balance in Code-Switching Speech [60.296574524609575]
  Large foundational models still struggle against code-switching test cases.<n>We use differentiable surrogates to mitigate context bias during generation.<n>Experiments with Arabic and Chinese-English showed that the models are able to predict the switching places more correctly.
  arXiv  Detail & Related papers  (2025-10-21T15:23:55Z)
• Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models [9.543689542888599]
  We propose an explainability-based evaluation procedure for vulnerability detectors.<n>Our approach, defined as Detection Alignment (DA), quantifies the agreement between the input source code lines.<n>We show how the predictions of such models are consistently biased by non-vulnerable lines.
  arXiv  Detail & Related papers  (2025-10-13T09:34:40Z)
• Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders [0.0]
  We apply sparse autoencoders to decompose Large Language Models, identifying directions that correspond to code correctness.<n>We find that code correctness directions in LLMs reliably predict incorrect code, while correction capabilities, though statistically significant, involve tradeoffs between fixing errors and preserving correct code.<n>Our mechanistic insights suggest three practical applications: prompting strategies should prioritize test examples over elaborate problem descriptions, predictor directions can serve as error alarms for developer review, and these same predictors can guide selective steering, intervening only when errors are anticipated to prevent the code corruption from constant steering.
  arXiv  Detail & Related papers  (2025-10-03T11:44:21Z)
• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Adaptive Self-supervision Algorithms for Physics-informed Neural Networks [59.822151945132525]
  Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function. We study the impact of the location of the collocation points on the trainability of these models. We propose a novel adaptive collocation scheme which progressively allocates more collocation points to areas where the model is making higher errors.
  arXiv  Detail & Related papers  (2022-07-08T18:17:06Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.