SLASH the Sink: Sharpening Structural Attention Inside LLMs

• URL: http://arxiv.org/abs/2605.10503v3
• Date: Mon, 18 May 2026 16:17:06 GMT
• Title: SLASH the Sink: Sharpening Structural Attention Inside LLMs
• Abstract summary: Large Language Models (LLMs) show remarkable semantic understanding but struggle with structural understanding when processing graph topologies in a serialized format.<n>We propose StructuraL Attention SHarpening (SLASH), which amplifies this internal structural understanding via a plug-and-play attention redistribution.
• Score: 55.75072688988806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) show remarkable semantic understanding but often struggle with structural understanding when processing graph topologies in a serialized format. Existing solutions rely on training external graph-based adapters or fine-tuning, which incur high costs and lost generalizability. In this work, we investigate the internal mechanisms of LLMs and present a critical finding: LLMs spontaneously reconstruct the graph's topology internally, evidenced by a distinct "sawtooth" pattern in their attention maps that structurally aligns with the "token-level adjacency matrix". However, this intrinsic structural understanding is diluted by the attention sink. We theoretically formalize this dilution as a representation bottleneck, stemming from a fundamental conflict: the model's anisotropic bias, essential for language tasks, suppresses the topology-aware local aggregation required for graph reasoning. To address this, we propose a training-free solution, named StructuraL Attention SHarpening (SLASH), which amplifies this internal structural understanding via a plug-and-play attention redistribution. Experiments on pure graph tasks and molecular prediction validate that SLASH delivers significant and consistent performance gains across diverse LLMs.

Related papers

• Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B [51.10897657510313]
  Large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction.<n>We investigate Llama 3.1-8B from a mechanistic interpretability point of view.<n>We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure.
  arXiv  Detail & Related papers  (2026-08-19T01:09:51Z)
• Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations [36.45654492179688]
  We find that hallucinations arise from systematic internal dynamics rather than random noise.<n>First, attention disproportionately concentrates toward shortcut-like structural cues rather than distributing across the full context.<n>Second, feed-forward representations fail to ground the provided knowledge, causing the model to revert to parametric memory.
  arXiv  Detail & Related papers  (2026-05-25T22:08:59Z)
• Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs [100.02824137397464]
  We investigate how Large Language Models adapt their internal representations when encountering inputs of increasing difficulty.<n>We reveal a consistent and quantifiable phenomenon: as task difficulty increases, the last hidden states of LLMs become substantially sparser.<n>This sparsity--difficulty relation is observable across diverse models and domains.
  arXiv  Detail & Related papers  (2026-03-03T18:48:15Z)
• Improving LLM Reasoning with Homophily-aware Structural and Semantic Text-Attributed Graph Compression [55.51959317490934]
  Large language models (LLMs) have demonstrated promising capabilities in Text-Attributed Graph (TAG) understanding.<n>We argue that graphs inherently contain rich structural and semantic information, and that their effective exploitation can unlock potential gains in LLMs reasoning performance.<n>We propose Homophily-aware Structural and Semantic Compression for LLMs (HS2C), a framework centered on exploiting graph homophily.
  arXiv  Detail & Related papers  (2026-01-13T03:35:18Z)
• When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected [10.031229573133709]
  Large language models (LLMs) have excelled at understanding natural language and integrating cross-modal signals.<n>Recent work has explored how different strategies for encoding graph structure affect LLM performance on text-attributed graphs.<n>We show that explicit structural priors are often unnecessary and, in some cases, counterproductive when powerful language models are involved.
  arXiv  Detail & Related papers  (2025-11-20T19:34:58Z)
• Learning Linear Regression with Low-Rank Tasks in-Context [8.347662730632047]
  In-context learning (ICL) is a key building block of modern large language models.<n>We analyze a linear attention model trained on low-rank regression tasks.<n>We find that statistical fluctuations in finite pre-training data induce an implicit regularization.
  arXiv  Detail & Related papers  (2025-10-06T07:27:49Z)
• Explaining Fine Tuned LLMs via Counterfactuals A Knowledge Graph Driven Framework [13.533352973355013]
  Low-Rank Adaptation (LoRA) has enabled large language models to acquire domain-specific knowledge with remarkable efficiency.<n>This work introduces a novel framework that explains fine-tuned LLMs via counterfactuals grounded in knowledge graphs.
  arXiv  Detail & Related papers  (2025-09-25T14:37:40Z)
• GRIL: Knowledge Graph Retrieval-Integrated Learning with Large Language Models [59.72897499248909]
  We propose a novel graph retriever trained end-to-end with Large Language Models (LLMs)<n>Within the extracted subgraph, structural knowledge and semantic features are encoded via soft tokens and the verbalized graph, respectively, which are infused into the LLM together.<n>Our approach consistently achieves state-of-the-art performance, validating the strength of joint graph-LLM optimization for complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-09-20T02:38:00Z)
• Beyond the Linear Separability Ceiling: Aligning Representations in VLMs [2.561649173827544]
  We introduce a diagnostic framework centered on the Linear Separability Ceiling (LSC)<n>We uncover a pervasive "alignment gap", where most models fail to generatively outperform their own representations.<n>By augmenting standard next-token prediction with a contrastive objective, our fine-tuning method activates dormant reasoning pathways.
  arXiv  Detail & Related papers  (2025-07-10T09:23:32Z)
• Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data [10.907949155931474]
  We study how large language models (LLMs) process graph-structured data.<n>We uncover unique phenomena regarding how LLMs apply attention to graph-structured data.<n>We analyze these findings to improve the modeling of such data by LLMs.
  arXiv  Detail & Related papers  (2025-05-04T14:40:31Z)
• Hallucination Detection in LLMs with Topological Divergence on Attention Graphs [60.83579255387347]
  Hallucination, i.e., generating factually incorrect content, remains a critical challenge for large language models.<n>We introduce TOHA, a TOpology-based HAllucination detector in the RAG setting.
  arXiv  Detail & Related papers  (2025-04-14T10:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.