The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs

• URL: http://arxiv.org/abs/2608.27750v1
• Date: Thu, 27 Aug 2026 22:26:25 GMT
• Title: The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs
• Abstract summary: We study the effectiveness of using linear probes to detect incorrect tool-calls in large language models (LLMs)<n>We find that probing is an effective means to catch a range of different tool-calling errors, including errors arising from using an argument that has the wrong value but the correct type.<n>We also show that probes are capable of generalizing to novel types of errors, which is critical in real world deployments.
• Score: 6.513364055484164
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The hidden states of large language models (LLMs) are known to capture rich information relating to model knowledge and behavior that can be hard to extract from examination of input and output alone. As LLM-based systems increasingly interface with the external world, one area of concern is detecting incorrect or improper use of tools. Motivated by this, we study the effectiveness of using linear probes to detect incorrect tool-calls, measuring probe efficacy across 18 tool-calling LLMs evaluated on the Berkeley Function Calling Leaderboard. Overall, we find that probing is an effective means to catch a range of different tool-calling errors, including errors arising from using an argument that has the wrong value but the correct type, which might not be recorded by standard logging frameworks. Important factors in success include model size, probing layer, and model post-training type. We also show that probes are capable of generalizing to novel types of errors, which is critical in real world deployments.

Related papers

• Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use [47.29360932085394]
  Large language models (LLMs) increasingly act as autonomous agents that must decide when to answer directly vs. when to invoke external tools.<n>We introduce a model-adaptive definition of tool-necessity, grounded in each model's empirical performance.<n>We find substantial mismatches of 26.5-54.0% and 30.8-41.8%, respectively.
  arXiv  Detail & Related papers  (2026-05-13T18:59:28Z)
• Mechanistic Anomaly Detection via Functional Attribution [6.1937472685875]
  We implement a functional attribution problem for anomaly detection in neural networks.<n>For backdoors in vision models, our method achieves state-of-the-art detection on BackdoorBench.<n>Our results establish functional attribution as an effective, modality-agnostic tool for detecting anomalous behavior in deployed models.
  arXiv  Detail & Related papers  (2026-04-21T01:39:57Z)
• ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration [68.89572566071575]
  ETAgent is a training framework for calibrating agent's tool-use behavior.<n>It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors.
  arXiv  Detail & Related papers  (2026-01-11T11:05:26Z)
• Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing [12.835224376066769]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their deployment is frequently undermined by undesirable behaviors.<n>We introduce a novel and efficient framework that diagnoses a range of undesirable LLM behaviors by analyzing representation and its gradients.<n>We systematically evaluate our method for tasks that include tracking harmful content, detecting backdoor poisoning, and identifying knowledge contamination.
  arXiv  Detail & Related papers  (2025-09-26T12:07:47Z)
• DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models [60.713908578319256]
  We propose Direct Discrepancy Learning (DDL) to optimize the detector with task-oriented knowledge.<n>Built upon this, we introduce DetectAnyLLM, a unified detection framework that achieves state-of-the-art MGTD performance.<n>MIRAGE samples human-written texts from 10 corpora across 5 text-domains, which are then re-generated or revised using 17 cutting-edge LLMs.
  arXiv  Detail & Related papers  (2025-09-15T10:59:57Z)
• Towards Automated Error Discovery: A Study in Conversational AI [48.735443116662026]
  We introduce Automated Error Discovery, a framework for detecting and defining errors in conversational AI.<n>We also propose SEEED (Soft Clustering Extended-Based Error Detection), as an encoder-based approach to its implementation.
  arXiv  Detail & Related papers  (2025-09-13T14:53:22Z)
• Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs [23.538087984484207]
  Machine unlearning (MU) for large language models (LLMs) seeks to remove specific undesirable data or knowledge from a trained model.<n>We identify a new vulnerability post-unlearning: unlearning trace detection.<n>We demonstrate that unlearning traces can be detected with over 90% accuracy even under forget-irrelevant inputs.
  arXiv  Detail & Related papers  (2025-06-16T21:03:51Z)
• ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMs [77.79172008184415]
  TOOLSCAN is a new benchmark to identify error patterns in LLM output on tool-use tasks.<n>We show that even the most prominent LLMs exhibit these error patterns in their outputs.<n>Researchers can use these insights from TOOLSCAN to guide their error mitigation strategies.
  arXiv  Detail & Related papers  (2024-11-20T18:56:22Z)
• Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing [63.20133320524577]
  We show that editing a small subset of parameters can effectively modulate specific behaviors of large language models (LLMs)<n>Our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen.
  arXiv  Detail & Related papers  (2024-07-11T17:52:03Z)
• Anomaly Detection of Tabular Data Using LLMs [54.470648484612866]
  We show that pre-trained large language models (LLMs) are zero-shot batch-level anomaly detectors. We propose an end-to-end fine-tuning strategy to bring out the potential of LLMs in detecting real anomalies.
  arXiv  Detail & Related papers  (2024-06-24T04:17:03Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection [63.56136319976554]
  Large Language Models (LLMs) generate hallucinations, which can cause significant damage when deployed for mission-critical tasks. We propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. We empirically evaluate our method and existing zero-resource detection methods on two datasets.
  arXiv  Detail & Related papers  (2023-10-10T10:14:59Z)
• Out-Of-Bag Anomaly Detection [0.9449650062296822]
  Data anomalies are ubiquitous in real world datasets, and can have an adverse impact on machine learning (ML) systems. We propose a novel model-based anomaly detection method, that we call Out-of-Bag anomaly detection. We show our method can improve the accuracy and reliability of an ML system as data pre-processing step via a case study on home valuation.
  arXiv  Detail & Related papers  (2020-09-20T06:01:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.