LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks

• URL: http://arxiv.org/abs/2606.17579v1
• Date: Tue, 16 Jun 2026 06:30:30 GMT
• Title: LLM Features Can Hurt GNNs: Concatenation Interference on Homophilous Graph Benchmarks
• Abstract summary: Adding LLM-generated node features to graph neural networks (GNNs) is widely reported to improve accuracy on standard benchmarks.<n>We document a contrasting observation: when LLM features are introduced through pure input concatenation, they can systematically degrade accuracy on homophilous benchmarks where end-to-end pipelines succeed.
• Score: 10.283803346813047
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adding LLM-generated node features to graph neural networks (GNNs) is widely reported to improve accuracy on standard benchmarks. We document a contrasting observation: when LLM features are introduced through pure input concatenation (rather than joint training, distillation, or prompt-conditioning), they can systematically degrade accuracy on the same homophilous benchmarks where end-to-end LLM pipelines succeed. With an MLP backbone on the Planetoid public split and bag-of-words original features, concatenating SBERT-encoded GPT-4o-mini TAPE features reduces PubMed test accuracy by -17.0 +/- 0.3 pp and Cora by -4.3 +/- 0.6 pp (CiteSeer -0.6 +/- 0.8 pp, within seed noise). The drop attenuates as we relax each condition (GCN / GCNII / GAT backbones, random splits, smaller encoders) and reverses on medium-homophily WikiCS (+4.4 pp) and ogbn-arxiv (+11.7 pp). To predict when concatenation helps versus hurts, we report a simple measure of LLM-alone discriminability, Delta_sig. Across 9 datasets Delta_sig correlates with the concatenation cost more strongly than homophily at point estimate (r^2 = 0.38 vs. 0.06; N=9, bootstrap CIs overlap). The bootstrap-best change-point is tau = 13.8 pp, and the rule "Delta_sig <= tau predicts non-positive concat cost" classifies 7/9 datasets correctly; since 60% of bootstrap samples place tau in [5, 30] pp, we treat Delta_sig as an interpretive lens rather than a precision filter. A dimension-controlled ablation on PubMed places the LLM-feature drop between same-source PCA (-2.3 pp) and same-dim Gaussian noise (-37.3 pp), ruling out dimensionality and weight-decay artifacts. Nine PubMed configurations fit a power law |Delta_concat| proportional to (sqrt(d_l/n))^1.31 with r^2 = 0.97; the low-Delta_sig, small-n corner is exactly where the headline -17 pp PubMed deficit appears.

Related papers

• The Filter Metric is Safety-Critical: Phantom Advantages in Group-Relative RL under Shaped Rewards [1.667501390377158]
  Group-relative policy optimization can discard nocontrast rollout groups through dynamic sampling.<n>We identify and quantify a metricpredicate mismatch under composite shaped rewards.
  arXiv  Detail & Related papers  (2026-09-12T10:36:55Z)
• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning [70.71795707610961]
  method augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry.<n>Method attains the lowest final residual on six of eight PDE benchmarks.<n>Three-seed $L2$ and $H1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy.
  arXiv  Detail & Related papers  (2026-08-29T21:34:47Z)
• Sparse Mutual Information Graph Averaging for Improving Random Indexing Embeddings [0.0]
  Sparse word embedding pipelines can avoid dense co-occurrence matrix materialization, dense factorization, and gradient training while still relying on sparse global corpus statistics.<n>This paper studies Random Indexing (RI) vectors refined by weighted averaging on a sparse Positive Pointwise Mutual Information (PPMI) graph.
  arXiv  Detail & Related papers  (2026-08-06T08:07:18Z)
• LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation [48.83701310501069]
  Large language models (LLMs) can generate neural-network modifications, but unrestricted generation is often invalid or harmful.<n>This paper studies a narrower setting: improving a weak target model using a stronger same-family source model from a neural-network database.<n>We propose a source-guided candidate-generation protocol with non-source controls, source-conditioned candidates, and a no-LLM hp_copy ablation under equal evaluation budgets.
  arXiv  Detail & Related papers  (2026-07-06T23:59:18Z)
• Sequential Consensus for Multi-Agent LLM Debates: A Wald-SPRT compute governor with calibration-based failure detection [0.0]
  Multi-agent debate improves factuality and reasoning, but most recipes pick a fixed round count.<n>We adapt Wald's Sequential Probability Ratio Test (SPRT) as a plug-in compute governor for LLM debates.<n>On GSM8K the rule stops in 1.01 average rounds (4.06 LLM calls) at 97.0% accuracy vs 99.0% for fixed-5 debate at 15 calls: a 3.7x call reduction at -2pp accuracy.<n>On MMLU the calibrated KL collapses to about 0 and the rule caps on 99.5% of items at 2.1x cost.
  arXiv  Detail & Related papers  (2026-05-18T23:43:12Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• When Domain Pretraining Interferes with Instruction Alignment: An Empirical Study of Adapter Merging in Medical LLMs [0.6345523830122167]
  Large language models can exhibit surprising adapter interference when combining domain adaptation and instruction alignment.<n>We study a two-stage LoRA pipeline for medical LLMs, where domain-oriented pre-training (PT) and supervised fine-tuning (SFT) are trained separately and later merged.
  arXiv  Detail & Related papers  (2026-01-26T10:54:06Z)
• Edge-aware baselines for ogbn-proteins in PyTorch Geometric: species-wise normalization, post-hoc calibration, and cost-accuracy trade-offs [45.88028371034407]
  We present reproducible, edge-aware baselines for ogbn-proteins in PyTorch Geometric.<n>Our strongest baseline is GraphSAGE with sum-based edge-to-node features.<n>Post-hoc per-label temperature scaling plus per-label thresholds substantially improves micro-F1 and expected calibration error (ECE) with negligible AUC change.
  arXiv  Detail & Related papers  (2025-11-17T11:09:46Z)
• Ensemble Threshold Calibration for Stable Sensitivity Control [0.0]
  We present an end-to-end framework that achieves exact recall with sub-percent variance over tens of millions of geometry pairs.<n>Our approach consistently hits a recall target within a small error, decreases redundant verifications relative to other calibrations, and runs end-to-end on a single TPU v3 core.
  arXiv  Detail & Related papers  (2025-10-02T15:22:28Z)
• Purge-Gate: Backpropagation-Free Test-Time Adaptation for Point Clouds Classification via Token Purging [33.86980091897939]
  Token Purging (PG) is a backpropagation-free approach that removes tokens highly affected by domain shifts before they reach attention layers.<n>We propose two variants: PG-SP, which leverages source statistics, and PG-SF, a fully source-free version relying on CLS-token-driven adaptation.<n>PG-SP achieves an average of +10.3% higher accuracy than state-of-the-art backpropagation-free methods, while PG-SF sets new benchmarks for source-free adaptation.
  arXiv  Detail & Related papers  (2025-09-11T18:33:40Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)
• Thinking Forward: Memory-Efficient Federated Finetuning of Language Models [21.438831528354513]
  Finetuning large language models (LLMs) in federated learning settings requires excessive memory for resource-constrained devices. In this paper, we introduce Spry, an FL algorithm that splits trainable weights of an LLM among participating clients. Spry achieves a low memory footprint, high accuracy, and fast convergence.
  arXiv  Detail & Related papers  (2024-05-24T13:37:48Z)
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch [72.97553348776425]
  We introduce DARE to set most delta parameters without affecting the abilities of Supervised Fine-Tuning (SFT) LMs. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT models and merge them into a single model. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open Leaderboard.
  arXiv  Detail & Related papers  (2023-11-06T13:43:07Z)
• Data-Free Dynamic Compression of CNNs for Tractable Efficiency [46.498278084317704]
  structured pruning approaches have shown promise in lowering floating-point operations without substantial drops in accuracy.<n>We propose HASTE (Hashing for Tractable Efficiency), a data-free, plug-and-play convolution module that instantly reduces a network's test-time inference cost without training or fine-tuning.<n>We demonstrate our approach on the popular vision benchmarks CIFAR-10 and ImageNet, where we achieve a 46.72% reduction in FLOPs with only a 1.25% loss in accuracy.
  arXiv  Detail & Related papers  (2023-09-29T13:09:40Z)
• Improving the Model Consistency of Decentralized Federated Learning [68.2795379609854]
  Federated Learning (FL) discards the central server and each client only communicates with its neighbors in a decentralized communication network. Existing DFL suffers from inconsistency among local clients, which results in inferior compared to FLFL. We propose DFedSAMMGS, where $1lambda$ is the spectral gossip matrix and $Q$ is the number of sparse data gaps.
  arXiv  Detail & Related papers  (2023-02-08T14:37:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.