When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More

• URL: http://arxiv.org/abs/2606.14476v1
• Date: Fri, 12 Jun 2026 14:13:19 GMT
• Title: When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More
• Abstract summary: A growing line of work equips large language model (LLM) agents with graph neural networks (GNNs) as callable tools.<n>We expose a frozen GNN to a ReAct-style LLM agent as an explicit tool and measure, on node classification over a text-attributed graph.<n>We find the agent does not exercise judgment: its predictions agree with the raw GNN's 97.6-99.2% of the time (5 seeds)<n>We find the cost of deference does not shrink as capability grows and grows where alternatives emerge.
• Score: 10.283803346813047
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A growing line of work equips large language model (LLM) agents with graph neural networks (GNNs) as callable tools, assuming the agent exercises judgment over when and how much to rely on such a tool. We test this directly. We expose a frozen GNN to a ReAct-style LLM agent as an explicit tool and measure, on node classification over a text-attributed graph (ogbn-arxiv, replicated on WikiCS), whether the agent uses the tool or merely obeys it. We find the agent does not exercise judgment: its predictions agree with the raw GNN's 97.6-99.2% of the time (5 seeds), collapsing into a GNN parrot that adopts the tool's output wholesale and bypasses its own reasoning. Sweeping backbone capability (Qwen2.5 0.5B-7B), the deference is not a weak-model artifact: among models able to invoke the tool, agreement rises with capability (0.60 to 0.98 from 1.5B to 7B). Crucially, the cost of deference does not shrink as capability grows and grows where alternatives emerge: a per-node oracle over the available actions beats the parrot by 0.09-0.18 at 3B and 0.12-0.22 at 7B, roughly doubling at high homophily, because the parrot is pinned to the frozen GNN while the agent's alternatives improve; at 7B a simple neighbour-label tool overtakes the GNN at high homophily (0.81 vs 0.71) yet the agent still defers. A simple selective-invocation gate recovers about half of that high-homophily gap (0.71 to 0.83) but yields no net global gain, and held-out estimates bound the best achievable gate over standard test-time features to at most a third of the oracle headroom: reliable selective invocation looks limited by available information, not merely router design. Our results are a cautionary measurement: evaluations of agent+tool systems cannot assume the agent adds judgment on top of the tool, and selective invocation must be designed in rather than expected to emerge from scale.

Related papers

• ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection [61.4148196085256]
  We present ROPE (Routed Origin Policy Enforcement), which is anchored in a structural notion of trust.<n>Our approach admits two provable guarantees: 1) at every step of a trajectory, no value whose only origin is attacker-writable content reaches an origin-guarded parameter, and 2) no rewording of an injection changes an admission decision.
  arXiv  Detail & Related papers  (2026-08-27T01:22:14Z)
• Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results [10.526208975530054]
  We study inference-time feed-forward network (FFN) intervention for improving structured outputs without retraining model weights.<n>Amplitude Gating (AG) is a non-destructive alternative that preserves pretrained FFN weight directions and modulates only activation magnitudes during generation.<n>These results identify tool-structured inference as the most credible first target for safe FFN-level inference optimization.
  arXiv  Detail & Related papers  (2026-07-13T07:28:55Z)
• Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI [2.2616066835313253]
  We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations.<n>We apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials.<n>The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly.
  arXiv  Detail & Related papers  (2026-07-03T02:20:04Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents [9.793491545168575]
  ToolGate predicts execute/skip decisions from trajectory text and simple structural features.<n>Across two Qwen3-VL backbones, ToolGate reduces token cost to 64-69% of the unrestricted ReAct baseline.
  arXiv  Detail & Related papers  (2026-06-02T02:44:27Z)
• Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains [20.11899219087138]
  We study two representative thinking with images'' agents, Thyme and DeepEyesV2, across real-world understanding, OCR, chart understanding, and mathematical reasoning.<n>We find that tool access yields little consistent aggregate improvement, does not reliably reduce generated-token cost, and leaves only a small tool-only solved set.
  arXiv  Detail & Related papers  (2026-06-01T15:04:25Z)
• Tool Calling is Linearly Readable and Steerable in Language Models [7.851713181228514]
  When a tool-calling agent picks the wrong tool, the failure is invisible until execution.<n>We use 12 instruction-tuned models across Gemma 3, Qwen 3, Qwen 2.5, and Llama 3.1.<n>We find the identity of the chosen tool is linearly and steerable inside the model.
  arXiv  Detail & Related papers  (2026-05-08T16:47:08Z)
• Beyond the Black Box: Interpretability of Agentic AI Tool Use [0.0]
  We introduce a mechanistic-interpretability toolkit built on Sparse Autoencoders and linear probes.<n>The framework reads model states before each action and infers both whether a tool is needed and how consequential the next tool action is likely to be.<n>We train the probes on multi-step trajectories from the NVIDIA Nemotron function-calling dataset and apply the same workflow to GPT-OSS 20B and Gemma 3 27B models.
  arXiv  Detail & Related papers  (2026-05-07T19:47:30Z)
• Scaling Test-Time Compute for Agentic Coding [126.72747643609274]
  We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories.<n>Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes.<n>Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0.
  arXiv  Detail & Related papers  (2026-04-16T17:39:33Z)
• In harmony with gpt-oss [0.0]
  No one has independently reproduced OpenAI's published scores for GPt-oss-20b with tools.<n>We reverse-engineered the model's in-distribution tools.<n>We then built a native harmony agent harness.
  arXiv  Detail & Related papers  (2026-04-01T01:16:13Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Agentic Test-Time Scaling for WebAgents [65.5178428849495]
  We present Confidence-Aware Test-Time Scaling (CATTS), which uses vote-derived uncertainty to allocate compute only when decisions are genuinely contentious.<n>CATTS improves performance on WebArena-Lite and GoBrowse by up to 9.1% over React while using up to 2.3x fewer tokens than uniform scaling.
  arXiv  Detail & Related papers  (2026-02-12T18:58:30Z)
• RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocations [52.752467948588816]
  We propose RSAgent, an agentic Multimodal Large Language Model (MLLM) which interleaves reasoning and action for segmentation via multi-turn tool invocations.<n> RSAgent queries a segmentation toolbox, observes visual feedback, and revises its spatial hypothesis using historical observations to re-localize targets and iteratively refine masks.<n>Extensive experiments show that RSAgent achieves a zero-shot performance of 66.5% gIoU on ReasonSeg test, improving over Seg-Zero-7B by 9%, and reaches 81.5% cIoU on RefCOCOg, demonstrating state-of-the-art performance
  arXiv  Detail & Related papers  (2025-12-30T06:50:11Z)
• ToolTweak: An Attack on Tool Selection in LLM-based Agents [52.17181489286236]
  We show that adversaries can systematically bias agents toward selecting specific tools, gaining unfair advantage over equally capable alternatives.<n>We present ToolTweak, a lightweight automatic attack that increases selection rates from a baseline of around 20% to as high as 81%.<n>To mitigate these risks, we evaluate two defenses: paraphrasing and perplexity filtering, which reduce bias and lead agents to select functionally similar tools more equally.
  arXiv  Detail & Related papers  (2025-10-02T20:44:44Z)
• Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning [93.30252692375886]
  Rule-based reinforcement learning can be used to enhance tool-calling in large language models.<n>Tool-N1-7B/14B clearly outperform GPT-4o on several major benchmarks.
  arXiv  Detail & Related papers  (2025-04-25T02:55:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.