Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration

• URL: http://arxiv.org/abs/2605.11591v1
• Date: Tue, 12 May 2026 06:17:01 GMT
• Title: Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration
• Abstract summary: Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval.<n>Yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance.<n>We propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction.
• Score: 36.91587844807066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval, yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance. Through empirical analysis, we identify a phenomenon termed Logit-Attention Divergence, in which output logits are heavily biased while internal attention maps remain well-aligned with relevant visual evidence. This observation reveals a fundamental limitation of existing logit-level calibration methods such as PriDe. Based on this insight, we propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction at inference time, requiring only a minimal calibration set with negligible computational overhead. Experiments on MS-COCO-based benchmarks show that our method substantially improves permutation invariance and achieves state-of-the-art performance, enhancing accuracy by over 40\% compared to baselines. Code is available at https://github.com/brightXian/LAD.

Related papers

• Localized Conformal Prediction for Image Classification with Vision-Language Models [5.212510010425201]
  We show that simple usage of the cosine similarity between test-time and calibration visual features is not sufficient to improve over the non-local baselines.<n>We propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction.
  arXiv  Detail & Related papers  (2026-06-30T12:34:40Z)
• Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding [56.38477453373368]
  LazyMCoT is a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty.<n>LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency.
  arXiv  Detail & Related papers  (2026-06-15T03:17:44Z)
• DevPrompt: Deviation-Based Prompt Learning for One-Normal ShotImage Anomaly Detection [0.0]
  Few-normal shot anomaly detection (FNSAD) aims to detect abnormal regions in images using only a few normal training samples.<n>Recent approaches leverage vision-language models such as CLIP with prompt-based learning to align image and text features.<n>We propose a deviation-guided prompt learning framework that integrates the semantic power of vision-language models with the statistical reliability of deviation-based scoring.
  arXiv  Detail & Related papers  (2026-01-21T20:35:51Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Average Calibration Losses for Reliable Uncertainty in Medical Image Segmentation [14.869379716339212]
  Deep neural networks for medical image segmentation are often overconfident, compromising both reliability and clinical utility.<n>We propose differentiable formulations of marginal L1 Average Error (mL1-ACE) as an auxiliary loss that can be computed on a per-image basis.<n>We find that the soft-binned variant yields the greatest improvements in calibration, over the Dice plus cross-entropy loss baseline, but often compromises segmentation performance.
  arXiv  Detail & Related papers  (2025-06-04T13:32:07Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Position: From Correlation to Causation: Max-Pooling-Based Multi-Instance Learning Leads to More Robust Whole Slide Image Classification [51.95824566163554]
  We argue that well-trained max-pooling-based MIL models can make predictions based on causal factors and avoid relying on spurious correlations.<n>We propose a simple yet effective max-pooling-based MIL method (FocusMIL) that outperforms existing mainstream attention-based methods on two datasets.
  arXiv  Detail & Related papers  (2024-08-18T12:15:22Z)
• Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration [60.95748658638956]
  This paper introduces the Multi-Label Confidence task, aiming to provide well-calibrated confidence scores in multi-label scenarios. Existing single-label calibration methods fail to account for category correlations, which are crucial for addressing semantic confusion. We propose the Dynamic Correlation Learning and Regularization algorithm, which leverages multi-grained semantic correlations to better model semantic confusion.
  arXiv  Detail & Related papers  (2024-07-09T13:26:21Z)
• Calibrating Segmentation Networks with Margin-based Label Smoothing [19.669173092632]
  We provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses. These losses could be viewed as approximations of a linear penalty imposing equality constraints on logit distances. We propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances.
  arXiv  Detail & Related papers  (2022-09-09T20:21:03Z)
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration [21.63888208442176]
  In spite of the dominant performances of deep neural networks, recent works have shown that they are poorly calibrated. We provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses. We propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances.
  arXiv  Detail & Related papers  (2021-11-30T14:21:47Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.