Uncertainty Quantification for LLM Function-Calling

• URL: http://arxiv.org/abs/2604.22985v1
• Date: Fri, 24 Apr 2026 19:56:39 GMT
• Title: Uncertainty Quantification for LLM Function-Calling
• Abstract summary: Large Language Models (LLMs) are increasingly deployed to autonomously solve real-world tasks.<n>Uncertainty Quantification (UQ) methods can be used to quantify this confidence and prevent potentially incorrect function calls.
• Score: 29.49069464022948
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly deployed to autonomously solve real-world tasks. A key ingredient for this is the LLM Function-Calling paradigm, a widely used approach for equipping LLMs with tool-use capabilities. However, an LLM calling functions incorrectly can have severe implications, especially when their effects are irreversible, e.g., transferring money or deleting data. Hence, it is of paramount importance to consider the LLM's confidence that a function call solves the task correctly prior to executing it. Uncertainty Quantification (UQ) methods can be used to quantify this confidence and prevent potentially incorrect function calls. In this work, we present what is, to our knowledge, the first evaluation of UQ methods for LLM Function-Calling (FC). While multi-sample UQ methods, such as Semantic Entropy, show strong performance for natural language Q&A tasks, we find that in the FC setting, it offers no clear advantage over simple single-sample UQ methods. Additionally, we find that the particularities of FC outputs can be leveraged to improve the performance of existing UQ methods in this setting. Specifically, multi-sample UQ methods benefit from clustering FC outputs based on their abstract syntax tree parsing, while single-sample UQ methods can be improved by selecting only semantically meaningful tokens when calculating logit-based uncertainty scores.

Related papers

• Evaluating Uncertainty Quantification Methods in Argumentative Large Language Models [24.97354151540176]
  We conduct experiments to evaluate ArgLLMs' performance on claim verification tasks when using different UQ methods.<n>Our results demonstrate that, despite its simplicity, direct prompting is an effective UQ strategy in ArgLLMs.
  arXiv  Detail & Related papers  (2025-09-26T19:59:09Z)
• On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization [54.965787768076254]
  Large Language Models have been recently exploited as judges for complex natural language processing tasks, such as Q&A.<n>We study the effectiveness of LLMs-as-a-judge for two code-related tasks, namely code generation and code summarization.
  arXiv  Detail & Related papers  (2025-07-22T13:40:26Z)
• UQLM: A Python Package for Uncertainty Quantification in Large Language Models [0.0]
  We introduce UQLM, a Python package for hallucination detection using state-of-the-art uncertainty quantification (UQ) techniques.<n>This toolkit offers a suite of UQ-based scorers that compute response-level confidence scores ranging from 0 to 1.
  arXiv  Detail & Related papers  (2025-07-08T17:22:32Z)
• Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results [10.551985027162576]
  Evaluations often use metrics like AUROC to assess how well UQ methods correlate with task correctness functions.<n>We show that mutual biases--when both UQ methods and correctness functions are biased by the same factors--systematically distort evaluation.<n>We identify LM-as-a-judge methods as the least length-biased, offering a promising path for a fairer UQ evaluation.
  arXiv  Detail & Related papers  (2025-04-18T13:13:42Z)
• CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought [10.166370877826486]
  Large language models (LLMs) excel in many tasks but struggle to accurately quantify uncertainty in their generated responses.<n>Existing uncertainty quantification (UQ) methods for LLMs are primarily prompt-wise rather than response-wise.<n>We propose CoT-UQ, a response-wise UQ framework that integrates LLMs' inherent reasoning capabilities through Chain-of-Thought.
  arXiv  Detail & Related papers  (2025-02-24T14:48:06Z)
• Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models [76.17975723711886]
  Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs)<n>In this work, we adapt Mahalanobis Distance (MD) - a well-established UQ technique in classification tasks - for text generation.<n>Our method extracts token embeddings from multiple layers of LLMs, computes MD scores for each token, and uses linear regression trained on these features to provide robust uncertainty scores.
  arXiv  Detail & Related papers  (2025-02-20T10:25:13Z)
• Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency [66.96286531087549]
  Uncertainty quantification (UQ) methods for Large Language Models (LLMs) encompass a variety of approaches.<n>We propose a novel approach to integrating model confidence with output consistency, resulting in a family of efficient and robust UQ methods.<n>We evaluate our approach across various tasks such as question answering, abstractive summarization, and machine translation.
  arXiv  Detail & Related papers  (2025-02-07T14:30:12Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• LaGR-SEQ: Language-Guided Reinforcement Learning with Sample-Efficient Querying [71.86163159193327]
  Large language models (LLMs) have recently demonstrated their impressive ability to provide context-aware responses via text. This ability could potentially be used to predict plausible solutions in sequential decision making tasks pertaining to pattern completion. We introduce LaGR, which uses this predictive ability of LLMs to propose solutions to tasks that have been partially completed by a primary reinforcement learning (RL) agent.
  arXiv  Detail & Related papers  (2023-08-21T02:07:35Z)
• SatLM: Satisfiability-Aided Language Models Using Declarative Prompting [68.40726892904286]
  We propose a new satisfiability-aided language modeling (SatLM) approach for improving the reasoning capabilities of large language models (LLMs) We use an LLM to generate a declarative task specification rather than an imperative program and leverage an off-the-shelf automated theorem prover to derive the final answer. We evaluate SATLM on 8 different datasets and show that it consistently outperforms program-aided LMs in the imperative paradigm.
  arXiv  Detail & Related papers  (2023-05-16T17:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.