Attribution-Guided Decoding

• URL: http://arxiv.org/abs/2509.26307v1
• Date: Tue, 30 Sep 2025 14:21:40 GMT
• Title: Attribution-Guided Decoding
• Abstract summary: We introduce Attribution-Guided Decoding (AGD), an interpretability-based decoding strategy.<n>Instead of directly manipulating model activations, AGD considers a set of high-probability output token candidates.<n>We demonstrate AGD's efficacy across three challenging domains.
• Score: 24.52258081219335
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The capacity of Large Language Models (LLMs) to follow complex instructions and generate factually accurate text is critical for their real-world application. However, standard decoding methods often fail to robustly satisfy these requirements, while existing control techniques frequently degrade general output quality. In this work, we introduce Attribution-Guided Decoding (AGD), an interpretability-based decoding strategy. Instead of directly manipulating model activations, AGD considers a set of high-probability output token candidates and selects the one that exhibits the highest attribution to a user-defined Region of Interest (ROI). This ROI can be flexibly defined over different parts of the model's input or internal components, allowing AGD to steer generation towards various desirable behaviors. We demonstrate AGD's efficacy across three challenging domains. For instruction following, we show that AGD significantly boosts adherence (e.g., improving the overall success rate on Llama 3.1 from 66.0% to 79.1%). For knowledge-intensive tasks, we show that guiding generation towards usage of internal knowledge components or contextual sources can reduce hallucinations and improve factual accuracy in both closed-book and open-book settings. Furthermore, we propose an adaptive, entropy-based variant of AGD that mitigates quality degradation and reduces computational overhead by applying guidance only when the model is uncertain. Our work presents a versatile, more interpretable, and effective method for enhancing the reliability of modern LLMs.

Related papers

• AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward [52.7128440950672]
  We propose AlphaGRPO to enhance multimodal generation capabilities without an additional cold-start stage.<n>Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks.<n>To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward)
  arXiv  Detail & Related papers  (2026-05-12T17:59:47Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval [60.25608870901428]
  Trustworthiness is a core research challenge for agentic AI systems built on Large Language Models (LLMs)<n>We propose the task of fact-checking without retrieval, focusing on the verification of arbitrary natural language claims, independent of their source robustness.
  arXiv  Detail & Related papers  (2026-03-05T18:42:51Z)
• Steering Language Models Before They Speak: Logit-Level Interventions [9.055997973281919]
  We propose a training-free inference-time logit intervention for controllable generation.<n>Our results show that statistically grounded logit steering can achieve large, consistent, and multi-task control gains.
  arXiv  Detail & Related papers  (2026-01-16T03:00:33Z)
• RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steering [62.63376387138257]
  We propose a plug-and-play intervention framework that adaptively steers large language models (LLMs) reasoning in activation space.<n>RISER constructs a library of reusable reasoning vectors and employs a lightweight Router to dynamically compose them for each input.<n>The Router is optimized via reinforcement learning under task-level rewards, activating latent cognitive primitives in an emergent and compositional manner.
  arXiv  Detail & Related papers  (2026-01-14T08:04:33Z)
• Representation Interventions Enable Lifelong Unstructured Knowledge Control [54.86207134539453]
  Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge.<n>We introduce RILKE, a robust and scalable method that treats knowledge control as interventions within the model's representation space.<n>During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference.<n>In inference, a query-adaptive router selects the appropriate module to guide the model's generation.
  arXiv  Detail & Related papers  (2025-11-25T22:15:00Z)
• EReLiFM: Evidential Reliability-Aware Residual Flow Meta-Learning for Open-Set Domain Generalization under Noisy Labels [85.78886153628663]
  Open-Set Domain Generalization aims to enable deep learning models to recognize unseen categories in new domains.<n>Label noise hinders open-set domain generalization by corrupting source-domain knowledge.<n>We propose Evidential Reliability-Aware Residual Flow Meta-Learning (EReLiFM) to bridge domain gaps.
  arXiv  Detail & Related papers  (2025-10-14T16:23:11Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• CAAD: Context-Aware Adaptive Decoding for Truthful Text Generation [31.469511576774252]
  We propose a context-aware adaptive decoding method for large language models.<n>Our approach achieves a 2.8 percent average improvement on TruthfulQA.<n>Our model-agnostic, scalable, and efficient method requires only a single generation pass.
  arXiv  Detail & Related papers  (2025-08-04T08:28:25Z)
• REAL: Reading Out Transformer Activations for Precise Localization in Language Model Steering [26.428347164111926]
  Inference-time steering aims to alter a large language model's responses without changing its parameters.<n>Existing approaches often rely on simplistic cues or ad hoc generalizations.<n>We introduce REAL, a framework for identifying behavior-relevant modules in Transformer models.
  arXiv  Detail & Related papers  (2025-06-10T02:16:50Z)
• KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG [63.82127103851471]
  Retrieval-Augmented Generation (RAG) enables large language models to access broader knowledge sources.<n>We demonstrate that enhancing generative models' capacity to process noisy content is equally critical for robust performance.<n>We present KARE-RAG, which improves knowledge utilization through three key innovations.
  arXiv  Detail & Related papers  (2025-06-03T06:31:17Z)
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models [57.20761595019967]
  We present Normalized Attention Guidance (NAG), an efficient, training-free mechanism that applies extrapolation in attention space with L1-based normalization and refinement.<n>NAG restores effective negative guidance where CFG collapses while maintaining fidelity.<n>NAG generalizes across architectures (UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image, video)
  arXiv  Detail & Related papers  (2025-05-27T13:30:46Z)
• UniErase: Towards Balanced and Precise Unlearning in Language Models [69.04923022755547]
  Large language models (LLMs) require iterative updates to address the outdated information problem.<n>UniErase is a novel unlearning framework that demonstrates precision and balanced performances between knowledge unlearning and ability retaining.
  arXiv  Detail & Related papers  (2025-05-21T15:53:28Z)
• Patterns and Mechanisms of Contrastive Activation Engineering [0.374490703387131]
  CAE has the potential to introduce a new paradigm of flexible, task-specific behavior tuning.<n>We analyze the performance of CAE in in in-distribution, out-of-distribution settings, evaluate drawbacks, and begin to develop comprehensive guidelines for its effective deployment.
  arXiv  Detail & Related papers  (2025-05-06T05:15:12Z)
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains [92.36624674516553]
  Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of large language models (LLMs)<n>We investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education.<n>We utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications.
  arXiv  Detail & Related papers  (2025-03-31T08:22:49Z)
• Towards LLM Guardrails via Sparse Representation Steering [11.710399901426873]
  Large Language Models (LLMs) have demonstrated remarkable performance in natural language generation tasks.<n>We propose a sparse encoding-based representation engineering method, named SRE, which decomposes polysemantic activations into a structured, monosemantic feature space.<n>By leveraging sparse autoencoding, our approach isolates and adjusts only task-specific sparse feature dimensions, enabling precise and interpretable steering of model behavior.
  arXiv  Detail & Related papers  (2025-03-21T04:50:25Z)
• Towards more Contextual Agents: An extractor-Generator Optimization Framework [0.0]
  Large Language Model (LLM)-based agents have demonstrated remarkable success in solving complex tasks across a wide range of general-purpose applications.<n>However, their performance often degrades in context-specific scenarios, such as specialized industries or research domains.<n>To address this challenge, our work introduces a systematic approach to enhance the contextual adaptability of LLM-based agents.
  arXiv  Detail & Related papers  (2025-02-18T15:07:06Z)
• Joint Localization and Activation Editing for Low-Resource Fine-Tuning [73.64004083269424]
  We propose a joint localization and activation editing (JoLA) method.<n>JoLA learns (1) which heads in the Transformer to edit (2) whether the intervention should be additive, multiplicative, or both and (3) the intervention parameters themselves.<n>We demonstrate that JoLA consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2025-02-03T09:13:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.