Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls

• URL: http://arxiv.org/abs/2608.03071v1
• Date: Tue, 04 Aug 2026 03:36:41 GMT
• Title: Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls
• Abstract summary: In domains such as cloud networking, even frontier models correctly complete fewer than half of tool calls.<n>We propose a unified probe-guided framework with two complementary approaches.<n>We show that our method substantially improves parameter generation, raising the average exact match from 19.7% to 59.6%.
• Score: 43.88014476513854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model agents derive much of their capability from tool use. Existing research on tool use has largely focused on selecting the right tool and orchestrating the order of calls. However, correctly filling the parameters of a tool call is equally critical for successful execution and has received far less attention. In domains such as cloud networking, even frontier models correctly complete fewer than half of tool calls. Inspired by recent analyses showing that LLM hidden states encode rich information about model predictions, we discover that while the model generates a parameter value, its hidden state contains a strong correctness signal: a simple linear probe can accurately predict whether the value will be correct. Based on this observation, we propose a unified probe-guided framework with two complementary approaches: probe-filtered bootstrapped training (PBT), which uses the probe to filter reliable self-generated calls for fine-tuning, and probe-guided reranking (PGR), which uses the probe to select better candidates during inference. To support systematic evaluation, we release ParamBench, a benchmark built from real cloud-network APIs that categorizes every instance into five difficulty levels according to parameter nesting depth, cross-parameter dependencies, and the reasoning required to derive values from earlier calls. Extensive experiments across 5 open models on ParamBench and 6 external benchmarks demonstrate that our method substantially improves parameter generation, raising the average exact match from 19.7% to 59.6%.

Related papers

• Learning Reasoning World Models for Parallel Code [26.139975890051925]
  A common remedy is to use coding agents that interact with external tools, but tool calls can be costly and sometimes impractical.<n>We propose Parallel-Code World Models (PCWMs) that aim to predict tool outcomes directly from parallel source code.<n>Our results suggest that reasoning models have the potential to serve as practical substitutes for external tool calls in parallel-coding agents.
  arXiv  Detail & Related papers  (2026-04-22T07:29:23Z)
• TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices [0.0]
  This paper investigates the effectiveness of small language models (SLMs) for agentic tasks (function/tool/API calling)<n>We describe parameter-driven optimization strategies that include supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), reinforcement learning (RL), and hybrid methods.<n>Our results demonstrate clear accuracy differences across model scales where medium-sized models (1-3B parameters) significantly outperform ultra-compact models (1B parameters)<n>This study highlights the importance of hybrid optimization strategies that enable small language models to deliver accurate, efficient, and stable agentic AI on edge devices.
  arXiv  Detail & Related papers  (2025-11-27T06:09:54Z)
• Maximally-Informative Retrieval for State Space Model Generation [59.954191072042526]
  We introduce Retrieval In-Context Optimization (RICO) to minimize model uncertainty for a particular query at test-time.<n>Unlike traditional retrieval-augmented generation (RAG), which relies on externals for document retrieval, our approach leverages direct feedback from the model.<n>We show that standard top-$k$ retrieval with model gradients can approximate our optimization procedure, and provide connections to the leave-one-out loss.
  arXiv  Detail & Related papers  (2025-06-13T18:08:54Z)
• SLOT: Sample-specific Language Model Optimization at Test-time [34.84741298539127]
  Large Language Models (LLMs) often struggle with complex instructions, leading to poor performances on those not well represented among general samples.<n>We propose SLOT, a novel and parameter-efficient test-time inference approach that enhances a language model's ability to more accurately respond to individual prompts.
  arXiv  Detail & Related papers  (2025-05-18T12:37:56Z)
• SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters [40.64474084442168]
  SimPER is an effective preference optimization algorithm for language model alignment.<n>SimPER is easy to implement and eliminates the need for expensive hyper parameter tuning and a reference model.<n>SimPER consistently and significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-02-02T19:25:41Z)
• ToolACE: Winning the Points of LLM Function Calling [139.07157814653638]
  ToolACE is an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data.<n>We demonstrate that models trained on our synthesized data, even with only 8B parameters, achieve state-of-the-art performance on the Berkeley Function-Calling Leaderboard.
  arXiv  Detail & Related papers  (2024-09-02T03:19:56Z)
• On the Effectiveness of Parameter-Efficient Fine-Tuning [79.6302606855302]
  Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. We show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. Despite the effectiveness of sparsity grounded by our theory, it still remains an open problem of how to choose the tunable parameters.
  arXiv  Detail & Related papers  (2022-11-28T17:41:48Z)
• Automating DBSCAN via Deep Reinforcement Learning [73.82740568765279]
  We propose a novel Deep Reinforcement Learning guided automatic DBSCAN parameters search framework, namely DRL-DBSCAN. The framework models the process of adjusting the parameter search direction by perceiving the clustering environment as a Markov decision process. The framework consistently improves DBSCAN clustering accuracy by up to 26% and 25% respectively.
  arXiv  Detail & Related papers  (2022-08-09T04:40:11Z)
• Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization [28.22184410167622]
  We present a data filtering method for open-domain dialogues. We score training samples with a quality measure, sort them in descending order, and filter out those at the bottom. Experimental results on two datasets show that our method can effectively identify untrustworthy samples.
  arXiv  Detail & Related papers  (2021-09-14T06:42:54Z)
• Efficient Nearest Neighbor Language Models [114.40866461741795]
  Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore. We show how to achieve up to a 6x speed-up in inference speed while retaining comparable performance.
  arXiv  Detail & Related papers  (2021-09-09T12:32:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.