Correcting Suppressed Log-Probabilities in Language Models with Post-Transformer Adapters

• URL: http://arxiv.org/abs/2604.14174v1
• Date: Wed, 25 Mar 2026 16:27:43 GMT
• Title: Correcting Suppressed Log-Probabilities in Language Models with Post-Transformer Adapters
• Abstract summary: Post-transformer adapter corrects suppression on 31 ideology-discriminating facts across Qwen3-4B, 8B, and 14B.<n> adapter memorizes all 15 training facts and generalizes to 11--39% of 16 held-out facts across 5 random splits per scale.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Alignment-tuned language models frequently suppress factual log-probabilities on politically sensitive topics despite retaining the knowledge in their hidden representations. We show that a 786K-parameter (approximately 0.02% of the base model) post-transformer adapter, trained on frozen hidden states, corrects this suppression on 31 ideology-discriminating facts across Qwen3-4B, 8B, and 14B. The adapter memorizes all 15 training facts and generalizes to 11--39% of 16 held-out facts across 5 random splits per scale, with zero knowledge regressions via anchored training. Both gated (SwiGLU) and ungated (linear bottleneck) adapters achieve comparable results; neither consistently outperforms the other (Fisher exact p > 0.09 at all scales). On instruct models, the adapter corrects log-probability rankings. When applied at all token positions during generation, the adapter produces incoherent output; however, when applied only at the current prediction position (last-position-only), the adapter produces coherent, less censored text. A logit-space adapter operating after token projection fails to produce coherent generation at any application mode, suggesting hidden-state intervention is the correct level for generation correction. A previously undocumented silent gradient bug in Apple MLX explains all null results in earlier iterations of this work: the standard pattern nn.value_and_grad(model, fn)(model.parameters()) returns zero gradients without error; the correct pattern nn.value_and_grad(model, fn)(model, data) resolves this. We provide a minimal reproduction and discuss implications for other adapter research using MLX.

Related papers

• ActMap: Single-Pass Uncertainty Quantification from Generation-Time Activation Maps [1.7403133838762448]
  ActMap is a representation that compresses the generation-time hidden-state trajectory into a fixed $12times32times128$ of temporal-statistic channels.<n>A lightweight classifier, instantiated as a compact Vision Transformer, reads an estimated correctness probability from each map in a fraction of a millisecond.
  arXiv  Detail & Related papers  (2026-09-10T13:04:24Z)
• Bayesian Wind Tunnels for Model Selection [0.4779196219827507]
  We show that transformers can perform exact Bayesian filtering within a fixed hypothesis class.<n>We then identify a sharp perceptual access condition: when the discnative statistic requires arithmetic, model selection succeeds with integer tokens.<n>A stationarity control confirms the operative factor: stable semantics, not integer identity, enable circuit compilation.
  arXiv  Detail & Related papers  (2026-07-01T04:05:08Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers [1.0152838128195467]
  We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck.<n>On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average.<n>Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested.
  arXiv  Detail & Related papers  (2026-03-16T01:29:26Z)
• Transcoder Adapters for Reasoning-Model Diffing [28.090312772463253]
  We introduce transcoder computation, a technique for learning an interpretable approximation.<n>We apply transcoder adapters to characterize the differences between Qwen2.5-Math-7B and its reasoning-distilled variant, DeepSeek-R1-Di-Qwen-7B.
  arXiv  Detail & Related papers  (2026-02-24T13:40:28Z)
• Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs [1.028001761831691]
  Self-interpretation methods prompt language models to describe their own internal states.<n>We show that training lightweight adapters on interpretability artifacts, while keeping the LM entirely frozen, yields reliable self-interpretation.
  arXiv  Detail & Related papers  (2026-02-10T22:50:02Z)
• PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise [60.63315470285562]
  MiniTruePrefixes is a novel specialized model that better detects factual inconsistencies over text prefixes.<n>We show that integrating MiniTruePrefixes into a controlled decoding framework substantially improves factual consistency in abstractive summarization.
  arXiv  Detail & Related papers  (2025-11-03T09:07:44Z)
• BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptation [30.435971066422706]
  We show that the discriminative performance of state-of-the-art CLIP adapters does not always correlate with their uncertainty estimation capabilities.<n>We introduce BayesAdapter, which leverages Bayesian inference to estimate a full probability distribution instead of a single point.<n>Our approach obtains high quality uncertainty estimates in the predictions, standing out in calibration and selective classification.
  arXiv  Detail & Related papers  (2024-12-12T20:48:06Z)
• Test-Time Model Adaptation with Only Forward Passes [68.11784295706995]
  Test-time adaptation has proven effective in adapting a given trained model to unseen test samples with potential distribution shifts. We propose a test-time Forward-Optimization Adaptation (FOA) method. FOA runs on quantized 8-bit ViT, outperforms gradient-based TENT on full-precision 32-bit ViT, and achieves an up to 24-fold memory reduction on ImageNet-C.
  arXiv  Detail & Related papers  (2024-04-02T05:34:33Z)
• PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees [95.87604231887353]
  Existing pFL methods introduce high communication and computation costs or are vulnerable to test communication. In PerAda, a parameter distillation and pFL pFL has superior performance, especially under test-time distribution. Our code is available at https://github.com/NV/PerAda.
  arXiv  Detail & Related papers  (2023-02-13T19:00:37Z)
• Revisiting adapters with adversarial training [39.018061382248085]
  We show that it is not necessary to separate batch statistics when co-training on clean and adversarial inputs. We show that training with adapters enables model soups through linear combinations of the clean and adversarial tokens.
  arXiv  Detail & Related papers  (2022-10-10T17:58:14Z)
• Thutmose Tagger: Single-pass neural model for Inverse Text Normalization [76.87664008338317]
  Inverse text normalization (ITN) is an essential post-processing step in automatic speech recognition. We present a dataset preparation method based on the granular alignment of ITN examples. One-to-one correspondence between tags and input words improves the interpretability of the model's predictions.
  arXiv  Detail & Related papers  (2022-07-29T20:39:02Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.