Evaluating the effectiveness of LLM-based interoperability

• URL: http://arxiv.org/abs/2510.23893v1
• Date: Mon, 27 Oct 2025 22:05:08 GMT
• Title: Evaluating the effectiveness of LLM-based interoperability
• Abstract summary: We analyze the effectiveness of large language models (LLMs) to make systems interoperate autonomously.<n>Qwen2.5-coder:32b was the most effective model using both strategies DIRECT and CODEGEN.
• Score: 0.3131740922192114
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Background: Systems of systems are becoming increasingly dynamic and heterogeneous, and this adds pressure on the long-standing challenge of interoperability. Besides its technical aspect, interoperability has also an economic side, as development time efforts are required to build the interoperability artifacts. Objectives: With the recent advances in the field of large language models (LLMs), we aim at analyzing the effectiveness of LLM-based strategies to make systems interoperate autonomously, at runtime, without human intervention. Method: We selected 13 open source LLMs and curated four versions of a dataset in the agricultural interoperability use case. We performed three runs of each model with each version of the dataset, using two different strategies. Then we compared the effectiveness of the models and the consistency of their results across multiple runs. Results: qwen2.5-coder:32b was the most effective model using both strategies DIRECT (average pass@1 >= 0.99) and CODEGEN (average pass@1 >= 0.89) in three out of four dataset versions. In the fourth dataset version, which included an unit conversion, all models using the strategy DIRECT failed, whereas using CODEGEN qwen2.5-coder:32b succeeded with an average pass@1 = 0.75. Conclusion: Some LLMs can make systems interoperate autonomously. Further evaluation in different domains is recommended, and further research on reliability strategies should be conducted.

Related papers

• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments [2.0144904127859236]
  Large Language Models (LLMs) have become integral to modern applications, yet their deployment remains challenging.<n>We introduce E2LLM, a framework designed to enable efficient LLM deployment in resource limited settings.
  arXiv  Detail & Related papers  (2026-06-02T15:23:28Z)
• SAGAI-MID: A Generative AI-Driven Middleware for Dynamic Runtime Interoperability [1.2744523252873352]
  SAGAI-MID is a FastAPI that uses large language models (LLMs) to dynamically detect and resolve schema mismatches at runtime.<n>We evaluate SAGAI-MID on 10 interoperability scenarios spanning version migration, IoT-to-analytics bridging, and protocol conversion across six LLMs from two providers.
  arXiv  Detail & Related papers  (2026-03-30T17:46:41Z)
• MAXS: Meta-Adaptive Exploration with LLM Agents [48.04723638253802]
  MaxS is a meta-adaptive reasoning framework based on Large Language Model (LLM) Agents.<n>MAXS employs a lookahead strategy to extend reasoning paths a few steps ahead.<n>It combines step consistency variance and inter-step trend slopes to jointly select stable, consistent, and high-value reasoning steps.
  arXiv  Detail & Related papers  (2026-01-14T07:48:00Z)
• Wisdom and Delusion of LLM Ensembles for Code Generation and Repair [45.969630994412846]
  We compare ten individual Large Language Models with three ensembles of these LLMs across three software engineering benchmarks.<n>We find that the theoretical upperbound for an ensemble's performance can be 83% above the best single model.<n>A diversity-based strategy realizes up to 95% of this theoretical potential, and proves effective even in small two-model ensembles.
  arXiv  Detail & Related papers  (2025-10-24T14:39:23Z)
• Teaching Language Models to Reason with Tools [73.21700643314917]
  We present emphHint-Engineering, a new data synthesis strategy that strategically injects diverse hints at optimal points within reasoning paths.<n>CoRT significantly enhances efficiency, reducing token usage by approximately 30% for the 32B model and 50% for the 1.5B model.
  arXiv  Detail & Related papers  (2025-10-23T08:41:44Z)
• Access Paths for Efficient Ordering with Large Language Models [7.826046892571884]
  We present the LLM ORDER BY operator as a logical abstraction and study its physical implementations within a unified evaluation framework.<n>We introduce three new designs: an agreement-based batch-size policy, a majority voting mechanism for pairwise sorting, and a two-way external merge sort adapted for LLMs.
  arXiv  Detail & Related papers  (2025-08-30T01:44:36Z)
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs [55.20230501807337]
  We present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework.<n>We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset.
  arXiv  Detail & Related papers  (2025-08-15T10:32:50Z)
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models [52.876185634349575]
  We propose to incorporate Mixture of Intra- and Inter-Modality Experts (MoIIE) to Large Vision-Language Models (LVLMs)<n>For each token, expert routing is guided by its modality, directing tokens to their respective intra-modality experts as well as a shared pool of inter-modality experts.<n>Our MoIIE models with 5.5B and 11.3B activated parameters match or even surpass the performance of existing advanced open-source MoE-LLMs based multi-modal models.
  arXiv  Detail & Related papers  (2025-08-13T13:00:05Z)
• YOLOv11-RGBT: Towards a Comprehensive Single-Stage Multispectral Object Detection Framework [23.587408383983163]
  We present YOLOv11-RGBT, a new comprehensive multimodal object detection framework.<n>We designed six multispectral fusion modes and successfully applied them to models from YOLOv3 to YOLOv12 and RT-DETR.<n>Experiments show our framework excels on three major open-source multispectral object detection datasets, like LLVIP and FLIR.
  arXiv  Detail & Related papers  (2025-06-17T16:37:00Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• Reinforced Model Merging [53.84354455400038]
  We present an innovative framework termed Reinforced Model Merging (RMM), which encompasses an environment and agent tailored for merging tasks.<n>By utilizing data subsets during the evaluation process, we addressed the bottleneck in the reward feedback phase, thereby accelerating RMM by up to 100 times.
  arXiv  Detail & Related papers  (2025-03-27T08:52:41Z)
• GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks [0.11458853556386796]
  This paper establishes a benchmark for evaluating tool-calling capabilities of large language models (LLMs)<n>We assess eight commercial LLMs (Claude Sonnet 3.5 and 4, Claude Haiku 3.5, Gemini 2.0 Flash, Gemini 2.5 Pro Preview, GPT-4o, GPT-4.1 and o4-mini) using a simple tool-calling agent equipped with 23 geospatial functions.<n>Results show o4-mini and Claude 3.5 Sonnet achieve the best overall performance, OpenAI's GPT-4.1, GPT-4o and Google's Gemini 2.5 Pro Preview do not fall far behind, but the last two are more efficient in
  arXiv  Detail & Related papers  (2025-03-23T16:20:14Z)
• Enhancing LLM Code Generation with Ensembles: A Similarity-Based Selection Approach [6.93983229112122]
  We propose an ensemble approach for large language models (LLMs) in code generation.<n>For voting, we compute syntactic and semantic similarity using CodeBLEU and behavioral equivalence.<n>We show through experiments that our ensemble approach consistently outperforms standalone LLMs.
  arXiv  Detail & Related papers  (2025-03-20T04:38:56Z)
• Decoding-Time Language Model Alignment with Multiple Objectives [116.42095026960598]
  Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. Here, we propose $textbfmulti-objective decoding (MOD)$, a decoding-time algorithm that outputs the next token from a linear combination of predictions. We show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method.
  arXiv  Detail & Related papers  (2024-06-27T02:46:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.