Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion

• URL: http://arxiv.org/abs/2604.18566v2
• Date: Tue, 21 Apr 2026 03:08:10 GMT
• Title: Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion
• Abstract summary: We present a systematic evaluation of large language model families -- spanning both proprietary cloud APIs and locally-hosted open-context models.<n>On CLD extraction, cloud models achieve 77--89% overall pass rates; the best local model reaches 77% (KimiK2.5GGUFQ3, zero-shot engine mid-tier cloud)<n>On Discussion Discussion, the best local models achieve 50--100% model building steps; only 0-50% error fixing prompts expose memory limits in local deployments.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a systematic evaluation of large language model families -- spanning both proprietary cloud APIs and locally-hosted open-source models -- on two purpose-built benchmarks for System Dynamics AI assistance: the \textbf{CLD Leaderboard} (53 tests, structured causal loop diagram extraction) and the \textbf{Discussion Leaderboard} (interactive model discussion, feedback explanation, and model building coaching). On CLD extraction, cloud models achieve 77--89\% overall pass rates; the best local model reaches 77\% (Kimi~K2.5~GGUF~Q3, zero-shot engine), matching mid-tier cloud performance. On Discussion, the best local models achieve 50--100\% on model building steps and 47--75\% on feedback explanation, but only 0--50\% on error fixing -- a category dominated by long-context prompts that expose memory limits in local deployments. A central contribution of this paper is a systematic analysis of \textit{model type effects} on performance: we compare reasoning vs.\ instruction-tuned architectures, GGUF (llama.cpp) vs.\ MLX (mlx\_lm) backends, and quantization levels (Q3 / Q4\_K\_M / MLX-3bit / MLX-4bit / MLX-6bit) across the same underlying model families. We find that backend choice has larger practical impact than quantization level: mlx\_lm does not enforce JSON schema constraints, requiring explicit prompt-level JSON instructions, while llama.cpp grammar-constrained sampling handles JSON reliably but causes indefinite generation on long-context prompts for dense models. We document the full parameter sweep ($t$, $p$, $k$) for all local models, cleaned timing data (stuck requests excluded), and a practitioner guide for running 671B--123B parameter models on Apple~Silicon.

Related papers

• Generating Workflow DAGs from Natural Language with Non-Reasoning LLMs [0.08537603362277842]
  We show that neuro-symbolic decomposition enables lower-cost, non-reasoning large language models to generate complex workflow DAGs.<n>We show that the method improves judge validity by 24 percentage points and achieves statistical equivalence to a reasoning model's out-of-the-box quality.
  arXiv  Detail & Related papers  (2026-08-31T05:01:29Z)
• AgentWeave: Routing Before Reasoning for Efficient Function Calling in Tool-Rich Language Models [0.0]
  We introduce AgentWeave, a deterministic pre-inference routing layer that constructs a bounded model-visible action space.<n>Relative to all-tools exposure, AgentWeave presents 70.18% fewer tools, uses 61.70% fewer input tokens, and exhibits 50.95% lower mean local-model latency.
  arXiv  Detail & Related papers  (2026-08-24T10:38:08Z)
• ModelLens: Finding the Best for Your Task from Myriads of Models [25.958674884209557]
  We introduce ModelLens, a unified framework for model recommendation in the wild.<n>Public leaderboard interactions, though scattered and noisy, collectively trace out an implicit atlas of model capabilities.<n>ModelLens ranks unseen models on unseen datasets without running candidates on the target dataset.
  arXiv  Detail & Related papers  (2026-05-08T00:49:05Z)
• Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models [0.0]
  LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures.<n>We benchmark three frontier LLMs on 100 natural-language questions over the Cleaned Contoso Retail dataset in ClickHouse.
  arXiv  Detail & Related papers  (2026-04-28T02:53:23Z)
• Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation [20.309826321619482]
  Large Language Models (LLMs) are increasingly adopted as automated judges for evaluating generated text.<n>We propose textbftextit OmniScore, a family of complementary, deterministic learned metrics.<n>We trained the models large-scale synthetic supervision ($sim$564k instances, in textbf107 languages) and evaluated using 8,617 manually annotated instances.
  arXiv  Detail & Related papers  (2026-04-06T18:36:54Z)
• Stacked from One: Multi-Scale Self-Injection for Context Window Extension [69.24689919827817]
  modelname is a novel framework based on multi-grained context compression and query-aware information acquisition.<n>modelnameachieves performance superior or comparable to strong baselines.
  arXiv  Detail & Related papers  (2026-03-05T03:16:16Z)
• One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking [7.856998585396422]
  Large language models (LLMs) are reshaping automated fact-checking (AFC) by enabling unified, end-to-end verification pipelines.<n>We propose textbfmulti-task learning (MTL) as a more efficient alternative that fine-tunes a single model to perform claim detection, evidence ranking, and stance detection jointly.
  arXiv  Detail & Related papers  (2026-01-16T13:44:25Z)
• MERGETUNE: Continued fine-tuning of vision-language models [77.8627788911249]
  Fine-tuning vision-language models (VLMs) often leads to catastrophic forgetting of pretrained knowledge.<n>We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted.
  arXiv  Detail & Related papers  (2026-01-15T15:15:53Z)
• HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions [50.61510609116118]
  HuggingR$4$ is a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection to efficiently select models.<n>It attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively.
  arXiv  Detail & Related papers  (2025-11-24T03:13:45Z)
• Are Prompts All You Need? Evaluating Prompt-Based Large Language Models (LLM)s for Software Requirements Classification [1.1458853556386799]
  This study tests whether prompt based large language models can reduce data needs.<n>We benchmark several models and prompting styles across multiple tasks on two English datasets, PROMISE and SecReq.
  arXiv  Detail & Related papers  (2025-09-17T09:58:26Z)
• Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing [64.38277118982698]
  Large Language Models (LLMs) have demonstrated human-like instruction-following abilities.<n>In this work, we explore how to route the best-performing LLM for each instruction to achieve better overall performance.<n>We develop a new paradigm, constructing capability instructions with model capability representation, user instruction, and performance inquiry prompts to assess the performance.
  arXiv  Detail & Related papers  (2025-02-24T16:10:53Z)
• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• DataComp-LM: In search of the next generation of training sets for language models [200.5293181577585]
  DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments with the goal of improving language models.<n>We provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations.<n>Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.
  arXiv  Detail & Related papers  (2024-06-17T17:42:57Z)
• Investigating Data Contamination in Modern Benchmarks for Large Language Models [27.479260572913724]
  Recent observations have underscored a disparity between the inflated benchmark scores and the actual performance of LLMs. We study data contamination by proposing two methods tailored for both open-source and proprietary LLMs. We find that certain commercial LLMs could surprisingly guess the missing option in various test sets.
  arXiv  Detail & Related papers  (2023-11-16T11:03:04Z)
• Prompt2Model: Generating Deployable Models from Natural Language Instructions [74.19816829003729]
  Large language models (LLMs) enable system builders to create competent NLP systems through prompting. In other ways, LLMs are a step backward from traditional special-purpose NLP models. We propose Prompt2Model, a general-purpose method that takes a natural language task description like the prompts provided to LLMs.
  arXiv  Detail & Related papers  (2023-08-23T17:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.