Attention Calibration for Position-Fair Dense Information Retrieval

• URL: http://arxiv.org/abs/2606.02737v1
• Date: Mon, 01 Jun 2026 18:04:26 GMT
• Title: Attention Calibration for Position-Fair Dense Information Retrieval
• Abstract summary: Dense retrieval models exhibit positional bias when relevant information appears later in a passage.<n>We ask whether this bias can be reduced at inference time, without retraining and without sacrificing overall retrieval effectiveness.<n>We adapt inference-time attention calibration to downstream retrieval and extend it with a strength coefficient that interpolates between the original and fully calibrated attention distributions.
• Score: 29.915108607793996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dense retrieval models exhibit positional bias: retrieval effectiveness degrades when relevant information appears later in a passage (Zeng et al., 2025). We ask whether this bias can be reduced at inference time, without retraining and without sacrificing overall retrieval effectiveness. To this end, we adapt inference-time attention calibration (Schuhmacher et al., 2026) to downstream retrieval and extend it with a strength coefficient lambda that interpolates between the original and fully calibrated attention distributions. Across three embedding models on SQuAD-PosQ and FineWeb-PosQ, we examine how basket size, calibrated layer set, and strength affect the trade-off between positional fairness and retrieval effectiveness, finding that partial calibration frequently outperforms full calibration. A single configuration (B=128, lambda=0.5, 50% layer depth) improves the harmonic mean of nDCG@10 across positional groups on FineWeb-PosQ for all three models without per-model tuning, and applies to both <s>-pooled and last-token-pooled architectures. This default configuration transfers without modification to PosIR, which spans 10 languages and 31 domains, reducing the Position Sensitivity Index in all 16 length-quartile x model x retrieval-setting combinations, while preserving or improving aggregate nDCG@10. We release our extended codebase at https://github.com/impresso/fair-sentence-transformers

Related papers

• Drift Calibration in Geometric Eye Tracking Systems [34.12106856838315]
  We present a calibration-focused dataset containing 163 trials from 12 participants.<n>We use it to evaluate global, local, and composite correction functions under a common spatial-extrapolation protocol.
  arXiv  Detail & Related papers  (2026-08-30T12:05:44Z)
• KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale [0.0]
  Kernel-based alignment of CLIP toward a vision centric teacher such as DINOv2 (KUEA) improves CLIP's visual representations while preserving text-encoder compatibility.<n>We introduce KALE, a loss-equilibration controller that tracks both losses and adaptively rescales the alignment weight toward a target ratio.<n>On a 3.3M-image CC12M subset, the aligned model preserves image-text retrieval and reproducibly improves SVHN linear probing.
  arXiv  Detail & Related papers  (2026-07-21T09:14:33Z)
• HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization [48.87937677659571]
  We introduce HARP (Hadamard-preconditioned Adaptive Rotation Processor), a learnable structured two-sided processor that replaces fixed Hadamard mixing.<n>Across 2-4 bit settings on models ranging from 1B to 70B parameters, HARP improves perplexity and zero-shot accuracy over fixed RHT.<n>HarP preserves deployment efficiency, reaching 128 tok/s versus 61 tok/s for FP16.
  arXiv  Detail & Related papers  (2026-05-28T12:24:15Z)
• Federated Martingale Posterior Samping [28.76805404959501]
  Federated Bayesian neural networks require fixing a prior on the model parameters together with a likelihood.<n>Motivated by the rapid progress of predictive models, the martingale posterior, also known as predictive Bayes, replaces the prior--likelihood pair with a predictive distribution.<n>This letter proposes federated martingale posterior (FMP) sampling, a one-shot embarrassingly parallel protocol in which each client uploads a small set of trainable data embeddings and the server runs the predictive sampler centrally.
  arXiv  Detail & Related papers  (2026-05-18T15:33:18Z)
• LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss [8.481133435038839]
  Large reasoning models reach competition-level math and coding accuracy via long autoregressive decoding.<n>Weight quantization is the standard tool for acceleration, but representative recipes lose accuracy on long-decoding reasoning benchmarks.<n>We propose LookAhead Quantization (LAQuant), a layer-wise weight-only QAT method that addresses both factors without online-transform overhead.
  arXiv  Detail & Related papers  (2026-05-09T07:35:38Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Bayesian Learning-Enhanced Navigation with Deep Smoothing for Inertial-Aided Navigation [8.510813384980262]
  BLENDS is a data-driven post-processing framework that augments the classical two-filter smoother with a transformer-based neural network.<n> BLENDS achieves horizontal position improvements of up to 63% over the baseline forward EKF.
  arXiv  Detail & Related papers  (2026-03-26T12:11:59Z)
• Correctness-Optimized Residual Activation Lens (CORAL): Transferrable and Calibration-Aware Inference-Time Steering [3.7758197704962835]
  We introduce CORAL, a regularized-time steering method that captures correctness signals from model internal activations using weight-decay probes.<n>CORAL consistently improves accuracy by 10% and expected calibration error (ECE) by 50% on average.<n>Our results support the hypothesis that distributed information in model internals can be extracted using regularized probes when individual neurons are insufficient.
  arXiv  Detail & Related papers  (2026-02-05T18:55:56Z)
• Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
  Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling. We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy. We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-13T14:13:49Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)
• Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks [54.23874144090228]
  A common approach is to learn a post-hoc calibration function that transforms the output of the original network into calibrated confidence scores. Previous post-hoc calibration techniques work only with simple calibration functions. We propose a new neural network architecture that represents a class of intra order-preserving functions.
  arXiv  Detail & Related papers  (2020-03-15T12:57:21Z)
• Calibrating Deep Neural Networks using Focal Loss [77.92765139898906]
  Miscalibration is a mismatch between a model's confidence and its correctness. We show that focal loss allows us to learn models that are already very well calibrated. We show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases.
  arXiv  Detail & Related papers  (2020-02-21T17:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.