Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation

• URL: http://arxiv.org/abs/2411.00412v1
• Date: Fri, 01 Nov 2024 07:18:31 GMT
• Title: Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation
• Authors: Bohan Lyu, Yadi Cao, Duncan Watson-Parris, Leon Bergen, Taylor Berg-Kirkpatrick, Rose Yu,
• Abstract summary: Large Language Models (LLMs) demonstrate promising capabilities in solving simple scientific problems. Human experts first assess problem complexity using domain knowledge before choosing an appropriate solution approach. We propose a novel two-component fine-tuning method. Our models demonstrate a 28.18% improvement in answer accuracy and a 13.89% increase in tool usage precision across all datasets.
• Score: 39.805610561281455
• License:
• Abstract: Large Language Models (LLMs) demonstrate promising capabilities in solving simple scientific problems but often produce hallucinations for complex ones. While integrating LLMs with tools can increase reliability, this approach typically results in over-reliance on tools, diminishing the model's ability to solve simple problems through basic reasoning. In contrast, human experts first assess problem complexity using domain knowledge before choosing an appropriate solution approach. Inspired by this human problem-solving process, we propose a novel two-component fine-tuning method. In the first component World Knowledge Distillation (WKD), LLMs learn directly from solutions generated using tool's information to internalize domain knowledge. In the second component Tool Usage Adaptation (TUA), we partition problems into easy and hard categories based on the model's direct answering accuracy. While maintaining the same alignment target for easy problems as in WKD, we train the model to intelligently switch to tool usage for more challenging problems. We validate our method on six scientific benchmark datasets, spanning mathematics, climate science and epidemiology. On average, our models demonstrate a 28.18% improvement in answer accuracy and a 13.89% increase in tool usage precision across all datasets, surpassing state-of-the-art models including GPT-4o and Claude-3.5.

Related papers

• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use. MeCo captures high-level cognitive signals in the representation space, guiding when to invoke tools. Our experiments show that MeCo accurately detects LLMs' internal cognitive signals and significantly improves tool-use decision-making.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving [1.5106583432923495]
  This paper introduces a novel evaluation technique, LLM-ProS, to assess the performance of state-of-the-art LLMs. Using a curated dataset of 166 World Finals problems from 2011 to 2024, we benchmark the models' reasoning, accuracy, and efficiency. Our results reveal significant differences in the models' abilities to generalize, adapt, and solve novel problems.
  arXiv  Detail & Related papers  (2025-02-04T18:55:14Z)
• Evaluating Language Models as Synthetic Data Generators [74.80905172696366]
  AgoraBench is a benchmark that provides standardized settings and metrics to evaluate LMs' data generation abilities. Through synthesizing 1.26 million training instances using 6 LMs and training 99 student models, we uncover key insights about LMs' data generation capabilities.
  arXiv  Detail & Related papers  (2024-12-04T19:20:32Z)
• Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents [27.112239616508834]
  Mixture of Refinement Agents (MoRA) is a novel agentic refinement framework for large language models (LLMs) MoRA iteratively refines the LLM generated base solution by correcting the aforementioned errors, resulting in a significant performance improvement for open-source LLMs. We evaluate our approach on the SciEval and MMLU subsets along with our own physics dataset (PhysicsQA)
  arXiv  Detail & Related papers  (2024-12-01T14:15:55Z)
• Deriva-ML: A Continuous FAIRness Approach to Reproducible Machine Learning Models [1.204452887718077]
  We show how data management tools can significantly improve the quality of data that is used for machine learning (ML) applications. We propose an architecture and implementation of such tools and demonstrate through two use cases how they can be used to improve ML-based eScience investigations.
  arXiv  Detail & Related papers  (2024-06-27T04:42:29Z)
• MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time [51.5039731721706]
  MindStar is a purely inference-based searching method for large language models. It formulates reasoning tasks as searching problems and proposes two search ideas to identify the optimal reasoning paths. It significantly enhances the reasoning abilities of open-source models, such as Llama-2-13B and Mistral-7B, and achieves comparable performance to GPT-3.5 and Grok-1.
  arXiv  Detail & Related papers  (2024-05-25T15:07:33Z)
• PETScML: Second-order solvers for training regression problems in Scientific Machine Learning [0.22499166814992438]
  In recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis. We introduce a software built on top of the Portable and Extensible Toolkit for Scientific computation to bridge the gap between deep-learning software and conventional machine-learning techniques.
  arXiv  Detail & Related papers  (2024-03-18T18:59:42Z)
• LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error [54.954211216847135]
  Existing large language models (LLMs) only reach a correctness rate in the range of 30% to 60%. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE) STE orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory.
  arXiv  Detail & Related papers  (2024-03-07T18:50:51Z)
• SciAgent: Tool-augmented Language Models for Scientific Reasoning [129.51442677710452]
  We introduce a new task setting named tool-augmented scientific reasoning. This setting supplements Large Language Models with scalable toolsets. We construct a tool-augmented training corpus named MathFunc which encompasses over 30,000 samples and roughly 6,000 tools. Building on MathFunc, we develop SciAgent to retrieve, understand and, if necessary, use tools for scientific problem solving.
  arXiv  Detail & Related papers  (2024-02-18T04:19:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.