Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystem

• URL: http://arxiv.org/abs/2606.29116v1
• Date: Sat, 27 Jun 2026 23:57:01 GMT
• Title: Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystem
• Abstract summary: Large Language Models (LLMs) are rapidly being adopted in low-code and no-code automation platforms.<n>We present the first large-scale empirical study of LLM agentic in low-code automation platforms.
• Score: 9.994953049271892
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) are rapidly being adopted in low-code and no-code automation platforms, where non-expert users design workflows that combine natural language understanding with external services and APIs. LLM agents are LLM systems that use LLMs as a core "brain" to reason, plan, and autonomously execute complex, multi-step tasks. In this paper, we present the first large-scale empirical study of LLM agentic workflows in low-code automation platforms. We analyze more than 6,000 publicly available n8n workflows and examine four aspects of their design: task distribution, structural and tool use patterns, reliability mechanisms, and autonomy levels. Our analysis shows that LLM workflows are not merely prompt response pipelines. Instead, LLMs are commonly embedded within broader automation structures involving control logic, external tools, communication services, storage systems, and human review points. We further find that while many workflows include lightweight post-processing or routing logic after LLM execution, explicit reliability mechanisms such as structured fallback paths, repair loops, failure-specific alerts, and human approval gates remain relatively uncommon. These results reveal a gap between the increasing deployment of LLM agents in practical automation ecosystems and the limited engineering support for reliability, safety, and governance. Overall, our study provides ten empirical findings and five research takeaways for researchers, platform developers, and practitioners seeking to understand and improve real-world LLM agentic workflows.

Related papers

• Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering [16.65912184999416]
  Large Language Model-based multi-agent systems are increasingly explored for software engineering tasks.<n>We present llmmas-otel, a lightweight and framework-agnostic tool that combines OpenTelemetry-based distributed tracing with fault injection.
  arXiv  Detail & Related papers  (2026-08-25T08:59:08Z)
• Benchmarking LLMs on File System Design and Implementation [3.7335834184334415]
  We present -Bench, an LLM benchmarking framework for fs-specific tasks.<n>We develop six types of tasks: basic understanding, basic implementation, performance modeling, debug, optimization, and new feature development.<n>With 505 tasks in -Bench, we conduct an empirical study with both open source (DeepSeek-V4-Flash, GLM-5.1, and MiniMax-M2.7) and proprietary (Claude-Opus-4.7, GPT-5.2, and Gemini-3.1-Pro) LLMs.<n>Our study discloses the model efficiency for different tasks, causes of failed f
  arXiv  Detail & Related papers  (2026-07-31T20:31:32Z)
• LLM4RTL: Tool-Assisted LLM for RTL Generation [5.338291061810534]
  Large language models (LLMs) have facilitated impressive progress in software engineering, code generation, tooling, and systems.<n>We propose a judge-renew-check-renew-check'' pipeline which updates a current public dataset using a hierarchy of state-of-the-art commercial LLM models.<n>Our experiments identify some common weaknesses of LLMs in rule-based reasoning and logic, and consequently, in RTL code-generation.
  arXiv  Detail & Related papers  (2026-06-13T23:10:40Z)
• Benchmarking Large Language Models for Multi-Language Software Vulnerability Detection [15.026084450436976]
  We present a study evaluating the performance of large language models (LLMs) on the software vulnerability detection task.<n>We have compiled a dataset comprising 8,260 vulnerable functions in Python, 7,505 in Java, and 28,983 in JavaScript.<n>These LLMs are benchmarked against five fine-tuned small language models and two open-source static application security testing tools.
  arXiv  Detail & Related papers  (2025-03-03T11:56:00Z)
• Practical Considerations for Agentic LLM Systems [5.455744338342196]
  This paper frames actionable insights and considerations from the research community in the context of established application paradigms.<n> Namely, we position relevant research findings into four broad categories--Planning, Memory Tools, and Control Flow--based on common practices in application-focused literature.
  arXiv  Detail & Related papers  (2024-12-05T11:57:49Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline.<n>Recent works have started exploiting large language models (LLM) to lessen such burden.<n>This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future [15.568939568441317]
  We investigate the current practice and solutions for large language models (LLMs) and LLM-based agents for software engineering.<n>In particular we summarise six key topics: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance.<n>We discuss the models and benchmarks used, providing a comprehensive analysis of their applications and effectiveness in software engineering.
  arXiv  Detail & Related papers  (2024-08-05T14:01:15Z)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence.<n>WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning [53.6472920229013]
  Large Language Models (LLMs) have demonstrated impressive capability in many natural language tasks. LLMs are prone to produce errors, hallucinations and inconsistent statements when performing multi-step reasoning. We introduce Q*, a framework for guiding LLMs decoding process with deliberative planning.
  arXiv  Detail & Related papers  (2024-06-20T13:08:09Z)
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
  Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
  arXiv  Detail & Related papers  (2024-06-19T00:28:58Z)
• Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents [56.822238860147024]
  Augmenting large language models with external tools has emerged as a promising approach to extend their utility.<n>Previous methods manually parse tool documentation and create in-context demonstrations, transforming tools into structured formats for LLMs to use in their step-by-step reasoning.<n>We propose AutoTools, a framework that enables LLMs to automate the tool-use workflow.
  arXiv  Detail & Related papers  (2024-05-26T11:40:58Z)
• MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programming [8.768061489034642]
  This paper presents Meaning-Typed Programming (MTP), a novel paradigm that automates integration through intuitive language-level constructs.<n>We implement MTP in Jac, a programming language that supersets Python, and find that MTP significantly reduces coding complexity while maintaining accuracy and efficiency.<n>Our user study shows that developers using MTP completed tasks 3.2x faster with 45% fewer lines of code compared to existing frameworks.
  arXiv  Detail & Related papers  (2024-05-14T21:12:01Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• AgentBench: Evaluating LLMs as Agents [99.12825098528212]
  Large Language Model (LLM) as agents has been widely acknowledged recently.<n>We present AgentBench, a benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
  arXiv  Detail & Related papers  (2023-08-07T16:08:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.