Good Tools are Half the Work: Tool Usage in Deep Learning Projects

• URL: http://arxiv.org/abs/2310.19124v2
• Date: Tue, 28 May 2024 16:13:22 GMT
• Title: Good Tools are Half the Work: Tool Usage in Deep Learning Projects
• Authors: Evangelia Panourgia, Theodoros Plessas, Ilias Balampanis, Diomidis Spinellis,
• Abstract summary: The rising popularity of deep learning (DL) methods and techniques has invigorated interest in the topic of SE4DL (Software Engineering for Deep Learning) About 63% of the GitHub repositories we examined contained at least one conventional SE tool. Software construction tools are the most widely adopted, while the opposite applies to management and maintenance tools.
• Score: 5.966029067108828
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rising popularity of deep learning (DL) methods and techniques has invigorated interest in the topic of SE4DL (Software Engineering for Deep Learning), the application of software engineering (SE) practices on deep learning software. Despite the novel engineering challenges brought on by the data-driven and non-deterministic paradigm of DL software, little work has been invested into developing DL-targeted SE tools. On the other hand, tools tackling non-SE issues specific to DL are actively used and referred to under the umbrella term "MLOps (Machine Learning Operations) tools". Nevertheless, the available literature supports the utility of conventional SE tooling in DL software development. Building upon previous mining software repositories (MSR) research on tool usage in open-source software works, we identify conventional and MLOps tools adopted in popular applied DL projects that use Python as the main programming language. About 63\% of the GitHub repositories we examined contained at least one conventional SE tool. Software construction tools are the most widely adopted, while the opposite applies to management and maintenance tools. Relatively few MLOps tools were found to be use, with only 20 tools out of a sample of 74 used in at least one repository. The majority of them were open-source rather than proprietary. One of these tools, TensorBoard, was found to be adopted in about half of the repositories in our study. Consequently, the widespread use of conventional SE tooling demonstrates its relevance to DL software. Further research is recommended on the adoption of MLOps tooling, focusing on the relevance of particular tool types, the development of required tools, as well as ways to promote the use of already available tools.

Related papers

• Chain of Tools: Large Language Model is an Automatic Multi-tool Learner [54.992464510992605]
  Automatic Tool Chain (ATC) is a framework that enables the large language models (LLMs) to act as a multi-tool user. To scale up the scope of the tools, we next propose a black-box probing method. For a comprehensive evaluation, we build a challenging benchmark named ToolFlow.
  arXiv  Detail & Related papers  (2024-05-26T11:40:58Z)
• What Are Tools Anyway? A Survey from the Language Model Perspective [67.18843218893416]
  Language models (LMs) are powerful yet mostly for text generation tasks. We provide a unified definition of tools as external programs used by LMs. We empirically study the efficiency of various tooling methods.
  arXiv  Detail & Related papers  (2024-03-18T17:20:07Z)
• LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error [54.954211216847135]
  Existing large language models (LLMs) only reach a correctness rate in the range of 30% to 60%. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE) STE orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory.
  arXiv  Detail & Related papers  (2024-03-07T18:50:51Z)
• EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction [56.02100384015907]
  EasyTool is a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction. It can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios.
  arXiv  Detail & Related papers  (2024-01-11T15:45:11Z)
• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use [82.24774504584066]
  Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. We introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools.
  arXiv  Detail & Related papers  (2023-10-04T19:39:26Z)
• Large Language Models as Tool Makers [85.00361145117293]
  We introduce a closed-loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs create their own reusable tools for problem-solving. Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks. 2) tool using: another LLM acts as the tool user, which applies the tool built by the tool maker for problem-solving.
  arXiv  Detail & Related papers  (2023-05-26T17:50:11Z)
• On the Tool Manipulation Capability of Open-source Large Language Models [19.6917640220883]
  We show can we enhance open-source LLMs to be competitive to leading closed LLM APIs in tool manipulation. Our techniques can boost leading open-source LLMs by up to 90% success rate, showing capabilities competitive to OpenAI GPT-4 in 4 out of 8 ToolBench tasks.
  arXiv  Detail & Related papers  (2023-05-25T22:10:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.