Use as Directed? A Comparison of Software Tools Intended to Check Rigor and Transparency of Published Work

• URL: http://arxiv.org/abs/2507.17991v1
• Date: Wed, 23 Jul 2025 23:49:28 GMT
• Title: Use as Directed? A Comparison of Software Tools Intended to Check Rigor and Transparency of Published Work
• Abstract summary: Lack of standardization and transparency in scientific reporting is a major problem.<n>There are several automated tools that have been designed to check different rigor criteria.<n>We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group.
• Score: 28.252424517077557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at https://github.com/PeterEckmann1/tool-comparison.

Related papers

• Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability [14.862168786900382]
  We introduce ToolBench-X, a benchmark for evaluating agents under recoverable reliability hazards.<n>Starting from clean tool environments, ToolBench-X injects five structured hazard types.<n>Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards.
  arXiv  Detail & Related papers  (2026-06-24T13:34:34Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains [20.11899219087138]
  We study two representative thinking with images'' agents, Thyme and DeepEyesV2, across real-world understanding, OCR, chart understanding, and mathematical reasoning.<n>We find that tool access yields little consistent aggregate improvement, does not reliably reduce generated-token cost, and leaves only a small tool-only solved set.
  arXiv  Detail & Related papers  (2026-06-01T15:04:25Z)
• When to Trust Tools? Adaptive Tool Trust Calibration For Tool-Integrated Math Reasoning [58.75883713573783]
  Tool-Integrated Reasoning has emerged as a promising paradigm that incorporates tool call and execution within the reasoning trajectory.<n>We find that when the reasoning of the model conflicts with the tool results, the model tends to believe in its own reasoning.<n>We introduce Adaptive Tool Trust (ATTC), a framework that guides the model to adaptively choose to trust or ignore the tool results.
  arXiv  Detail & Related papers  (2026-04-09T14:14:37Z)
• Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents [22.64138018985385]
  We argue that failures stem from both tool-use accuracy (how well an agent invokes a tool) and intrinsic tool accuracy (the tool's own correctness)<n>We introduce OpenTools, a community-driven toolbox that standardizes tool schemas.<n>OpenTools includes the core framework, an initial tool set, evaluation pipelines, and a contribution protocol.
  arXiv  Detail & Related papers  (2026-03-31T18:42:36Z)
• ToolMATH: A Math Tool Benchmark for Realistic Long-Horizon Multi-Tool Reasoning [11.99927786717109]
  ToolMATH turns math problems into a controlled, correctness-checkable benchmark with tool sets.<n>ToolMATH provides actionable diagnostic evidence of failure modes in tool-augmented agents.
  arXiv  Detail & Related papers  (2026-02-24T09:23:12Z)
• A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM [54.38424417079265]
  A Software Bill of Materials (SBOM) is a machine-readable artifact that organizes software information.<n>Following standards, organizations have developed tools for generating and utilizing SBOMs.<n>This paper presents the first large-scale, two-stage empirical analysis of the adherence gap, using our automated evaluation framework, SAP.
  arXiv  Detail & Related papers  (2026-01-09T08:26:05Z)
• ToolLibGen: Scalable Automatic Tool Creation and Aggregation for LLM Reasoning [80.10274552177096]
  Large Language Models (LLMs) equipped with external tools have demonstrated enhanced performance on complex reasoning tasks.<n>The widespread adoption of this tool-augmented reasoning is hindered by the scarcity of domain-specific tools.<n>We propose a systematic approach to automatically an unstructured collection of tools into a structured tool library.
  arXiv  Detail & Related papers  (2025-10-09T04:11:16Z)
• TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use [74.47746287181383]
  Large language model (LLM)-based agents increasingly rely on tool use to complete real-world tasks.<n>We introduce TRAJECT-Bench, a trajectory-aware benchmark to comprehensively evaluate LLMs' tool use capability.
  arXiv  Detail & Related papers  (2025-10-06T07:30:25Z)
• ToolTweak: An Attack on Tool Selection in LLM-based Agents [52.17181489286236]
  We show that adversaries can systematically bias agents toward selecting specific tools, gaining unfair advantage over equally capable alternatives.<n>We present ToolTweak, a lightweight automatic attack that increases selection rates from a baseline of around 20% to as high as 81%.<n>To mitigate these risks, we evaluate two defenses: paraphrasing and perplexity filtering, which reduce bias and lead agents to select functionally similar tools more equally.
  arXiv  Detail & Related papers  (2025-10-02T20:44:44Z)
• BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models [55.119657444627855]
  Large language models (LLMs) often rely on external tools drawn from marketplaces where multiple providers offer functionally equivalent options.<n>This raises a critical point concerning fairness: if selection is systematically biased, it can degrade user experience and distort competition.<n>We introduce a benchmark of diverse tool categories, each containing multiple functionally equivalent tools, to evaluate tool-selection bias.
  arXiv  Detail & Related papers  (2025-09-30T22:02:13Z)
• Tool Graph Retriever: Exploring Dependency Graph-based Tool Retrieval for Large Language Models [43.50789219459378]
  We propose Tool Graph Retriever (TGR), which exploits the dependencies among tools to learn better tool representations for retrieval.<n>First, we construct a dataset termed TDI300K to train a discriminator for identifying tool dependencies.<n>Then, we represent all candidate tools as a tool dependency graph and use graph convolution to integrate the dependencies into their representations.
  arXiv  Detail & Related papers  (2025-08-07T08:36:26Z)
• ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks [54.52092001110694]
  ThinkGeo is a benchmark designed to evaluate tool-augmented agents on remote sensing tasks via structured tool use and multi-step planning.<n>Inspired by tool-interaction paradigms, ThinkGeo includes human-curated queries spanning a wide range of real-world applications.<n>Our analysis reveals notable disparities in tool accuracy and planning consistency across models.
  arXiv  Detail & Related papers  (2025-05-29T17:59:38Z)
• Vexed by VEX tools: Consistency evaluation of container vulnerability scanners [0.0]
  This paper presents a study that analyzed state-of-the-art vulnerability scanning tools applied to containers.<n>We have focused the work on tools following the Vulnerability Exploitability eXchange (VEX) format.
  arXiv  Detail & Related papers  (2025-03-18T16:22:43Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform Setting [2.3818760805173342]
  We analyze a combination of three sentiment analysis tools in a voting classifier according to their reliability and performance.<n>The results indicate that this kind of combination of tools is a good choice in the within-platform setting.<n>However, a majority vote does not necessarily lead to better results when applying in cross-platform domains.
  arXiv  Detail & Related papers  (2025-02-10T16:51:51Z)
• Does the Tool Matter? Exploring Some Causes of Threats to Validity in Mining Software Repositories [9.539825294372786]
  We use two tools to extract and analyse ten large software projects.<n>Despite similar trends, even simple metrics such as the numbers of commits and developers may differ by up to 500%.<n>We find that such substantial differences are often caused by minor technical details.
  arXiv  Detail & Related papers  (2025-01-25T07:42:56Z)
• Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval [47.81307125613145]
  Re-Invoke is an unsupervised tool retrieval method designed to scale effectively to large toolsets without training. We employ a novel multi-view similarity ranking strategy based on intents to pinpoint the most relevant tools for each query. Our evaluation demonstrates that Re-Invoke significantly outperforms state-of-the-art alternatives in both single-tool and multi-tool scenarios.
  arXiv  Detail & Related papers  (2024-08-03T22:49:27Z)
• Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark [8.573278807410507]
  This paper presents a new tool learning dataset Seal-Tools. Seal-Tools contains self-instruct API-like tools. It also includes instances which demonstrate the practical application of tools.
  arXiv  Detail & Related papers  (2024-05-14T06:50:19Z)
• What Are Tools Anyway? A Survey from the Language Model Perspective [67.18843218893416]
  Language models (LMs) are powerful yet mostly for text generation tasks. We provide a unified definition of tools as external programs used by LMs. We empirically study the efficiency of various tooling methods.
  arXiv  Detail & Related papers  (2024-03-18T17:20:07Z)
• TOOLVERIFIER: Generalization to New Tools via Self-Verification [69.85190990517184]
  We introduce a self-verification method which distinguishes between close candidates by self-asking contrastive questions during tool selection. Experiments on 4 tasks from the ToolBench benchmark, consisting of 17 unseen tools, demonstrate an average improvement of 22% over few-shot baselines.
  arXiv  Detail & Related papers  (2024-02-21T22:41:38Z)
• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use [79.87054552116443]
  Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities.<n>We introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools.<n>We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools.
  arXiv  Detail & Related papers  (2023-10-04T19:39:26Z)
• A Comprehensive Study on Quality Assurance Tools for Java [15.255117038871337]
  Quality assurance (QA) tools are receiving more and more attention and are widely used by developers. Most existing research is limited in the following ways:. They compare tools without considering scanning rules analysis. They disagree on the effectiveness of tools due to the study methodology and benchmark dataset. There is no large-scale study on the analysis of time performance.
  arXiv  Detail & Related papers  (2023-05-26T10:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.