Belief or Circuitry? Causal Evidence for In-Context Graph Learning

• URL: http://arxiv.org/abs/2605.08405v1
• Date: Fri, 08 May 2026 19:11:19 GMT
• Title: Belief or Circuitry? Causal Evidence for In-Context Graph Learning
• Abstract summary: We show that either the model tracks global topology, or it copies local transitions.<n>We present two lines of evidence that neither account alone is sufficient.<n>Our findings are most consistent with a dual-mechanism account in which genuine structure inference and induction circuits operate in parallel.
• Score: 3.9356042574269545
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How do LLMs learn in-context? Is it by pattern-matching recent tokens, or by inferring latent structure? We probe this question using a toy graph random-walk across two competing graph structures. This task's answer is, in principle, decidable: either the model tracks global topology, or it copies local transitions. We present two lines of evidence that neither account alone is sufficient. First, reconstructing the internal representation structure via PCA reveals that at intermediate mixture ratios, both graph topologies are encoded in orthogonal principal subspaces simultaneously. This pattern is difficult to reconcile with purely local transition copying. Second, residual-stream activation patching and graph-difference steering causally intervene on this graph-family signal: late-layer patching almost fully transfers the clean graph preference, while linear steering moves predictions in the intended direction and fails under norm-matched and label-shuffled controls. Taken together, our findings are most consistent with a dual-mechanism account in which genuine structure inference and induction circuits operate in parallel.

Related papers

• From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers [53.505136935517925]
  Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood.<n>We prove that single-head attention (SHA) is exactly a connection propagation step with constant transport.<n>We find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory.
  arXiv  Detail & Related papers  (2026-07-12T09:44:54Z)
• Graph Cascades: Contagion-Based Mesoscopic Rewiring for Structure-Aware Graph Machine Learning [8.480205772461925]
  We introduce Graph Cascades, a mesoscopic rewiring strategy for Graph Neural Networks (GNNs) and Graph Transformers (GTs)<n>Using contagion-based diffusion processes, Graph Cascades constructs auxiliary graphs where node pairs supported by repeated multi-hop reinforcement are promoted to direct neighbors.<n> Empirically, across node-classification benchmarks, Graph Cascades improves multiple GNN and sparse-GT backbones, with the most reliable gains observed on heterophilic and moderate- to high-degree homophilic graphs.
  arXiv  Detail & Related papers  (2026-06-03T16:07:24Z)
• When Do Graph Foundation Models Transfer? A Data-Centric Theory [68.54337055801571]
  Graph foundation models (GFMs) aim to reuse a single backbone across diverse graph domains, yet their transfer is often uneven and can exhibit negative transfer.<n>We show that for both set-based and message-passing tokenizations, any Lipschitz backbone admits an explicit decomposition of cross-domain output shift.<n>A key ingredient is positional-encoding (PE) stability: we establish stability guarantees for spectral PEs and highlight contrasting behaviors of eigenvector- versus subspace-based PEs.
  arXiv  Detail & Related papers  (2026-05-28T12:07:27Z)
• Patch-Effect Graph Kernels for LLM Interpretability [0.0]
  Mechanistic interpretability aims to reverse-engineer transformer computations by identifying causal circuits through activation patching.<n>We propose a framework that reframes mechanistic analysis as a graph machine-learning problem by representing activation-patching profiles as patch-effect graphs over model components.
  arXiv  Detail & Related papers  (2026-05-07T16:03:47Z)
• Bridging the Divide: End-to-End Sequence-Graph Learning [47.95529678412846]
  We argue that sequences and graphs are not separate problems but complementary facets of the same dataset.<n>We introduce BRIDGE, a unified end-to-end architecture that couples a sequence encoder with a GNN under a single objective.<n>We show that BRIDGE consistently outperforms static GNNs, temporal graph methods, and sequence-only baselines on ranking and classification metrics.
  arXiv  Detail & Related papers  (2025-10-29T03:06:54Z)
• Inferred global dense residue transition graphs from primary structure sequences enable protein interaction prediction via directed graph convolutional neural networks [2.719135068309479]
  Accurate prediction of protein-protein interactions (PPIs) is crucial for understanding cellular functions and advancing drug development.<n>Existing in-silico methods use direct sequence embeddings from Protein Language Models (PLMs)<n>Others use Graph Neural Networks (GNNs) for 3D protein structures.<n>We introduce a novel framework for downstream PPI prediction through link prediction.
  arXiv  Detail & Related papers  (2025-10-15T22:15:31Z)
• Align-GRAG: Reasoning-Guided Dual Alignment for Graph Retrieval-Augmented Generation [79.75818239774952]
  Large language models (LLMs) have demonstrated remarkable capabilities, but still struggle with issues like hallucinations and outdated information.<n>Retrieval-augmented generation (RAG) addresses these issues by grounding LLM outputs in external knowledge with an Information Retrieval (IR) system.<n>We propose Align-GRAG, a novel reasoning-guided dual alignment framework in post-retrieval phrase.
  arXiv  Detail & Related papers  (2025-05-22T05:15:27Z)
• Automatically Identifying Local and Global Circuits with Linear Computation Graphs [45.760716193942685]
  We introduce our circuit discovery pipeline with Sparse Autoencoders (SAEs) and a variant called Transcoders. Our methods do not require linear approximation to compute the causal effect of each node. We analyze three kinds of circuits in GPT-2 Small: bracket, induction, and Indirect Object Identification circuits.
  arXiv  Detail & Related papers  (2024-05-22T17:50:04Z)
• You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Prediction [79.15394378571132]
  We investigate a previously overlooked phenomenon: in many cases, a densely connected, complementary graph can be found for the original graph. The denser graph may share nodes with the original graph, which offers a natural bridge for transferring selective, meaningful knowledge. We identify this setting as Graph Intersection-induced Transfer Learning (GITL), which is motivated by practical applications in e-commerce or academic co-authorship predictions.
  arXiv  Detail & Related papers  (2023-02-27T22:56:06Z)
• Counterfactual Intervention Feature Transfer for Visible-Infrared Person Re-identification [69.45543438974963]
  We find graph-based methods in the visible-infrared person re-identification task (VI-ReID) suffer from bad generalization because of two issues. The well-trained input features weaken the learning of graph topology, making it not generalized enough during the inference process. We propose a Counterfactual Intervention Feature Transfer (CIFT) method to tackle these problems.
  arXiv  Detail & Related papers  (2022-08-01T16:15:31Z)
• Graph Spectral Embedding using the Geodesic Betweeness Centrality [76.27138343125985]
  We introduce the Graph Sylvester Embedding (GSE), an unsupervised graph representation of local similarity, connectivity, and global structure. GSE uses the solution of the Sylvester equation to capture both network structure and neighborhood proximity in a single representation.
  arXiv  Detail & Related papers  (2022-05-07T04:11:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.