Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning

• URL: http://arxiv.org/abs/2502.08056v1
• Date: Wed, 12 Feb 2025 01:36:27 GMT
• Title: Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning
• Authors: Zijian He, Reyna Abhyankar, Vikranth Srivatsa, Yiying Zhang,
• Abstract summary: gen-AI that involve multiple ML model calls, tool/API calls, data retrieval, or generic code execution are often tuned manually in an ad-hoc way. AdaSeek organizes workflow tuning methods into different layers based on the user-specified total search budget. Cognify improves these workflow's generation quality by up to 2.8x, reduces execution monetary cost by up to 10x, and reduces end-to-end latency by 2.7x.
• Score: 6.328780056857816
• License:
• Abstract: Today's gen-AI workflows that involve multiple ML model calls, tool/API calls, data retrieval, or generic code execution are often tuned manually in an ad-hoc way that is both time-consuming and error-prone. In this paper, we propose a systematic approach for automatically tuning gen-AI workflows. Our key insight is that gen-AI workflows can benefit from structure, operator, and prompt changes, but unique properties of gen-AI workflows require new optimization techniques. We propose AdaSeek, an adaptive hierarchical search algorithm for autotuning gen-AI workflows. AdaSeek organizes workflow tuning methods into different layers based on the user-specified total search budget and distributes the budget across different layers based on the complexity of each layer. During its hierarchical search, AdaSeek redistributes the search budget from less useful to more promising tuning configurations based on workflow-level evaluation results. We implement AdaSeek in a workflow autotuning framework called Cognify and evaluate Cognify using six types of workflows such as RAG-based QA and text-to-SQL transformation. Overall, Cognify improves these workflows' generation quality by up to 2.8x, reduces execution monetary cost by up to 10x, and reduces end-to-end latency by 2.7x.

Related papers

• EvoFlow: Evolving Diverse Agentic Workflows On The Fly [21.82515160298748]
  EvoFlow is a niching evolutionary algorithm-based framework to automatically search a population of complexity and heterogeneous agentic. We show that EvoFlow can evolve a population ranging from simple I/O tasks to complex multi-turn interactions.
  arXiv  Detail & Related papers  (2025-02-11T08:48:46Z)
• ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization [51.280919773837645]
  We develop ScoreFlow, a high-performance framework for agent workflow optimization. ScoreFlow incorporates Score-DPO, a novel variant of the direct preference optimization method that accounts for quantitative feedback. It achieves an 8.2% improvement over existing baselines across question answering, coding, and mathematical reasoning.
  arXiv  Detail & Related papers  (2025-02-06T18:47:49Z)
• Flow: A Modular Approach to Automated Agentic Workflow Generation [53.073598156915615]
  Multi-agent frameworks powered by large language models (LLMs) have demonstrated great success in automated planning and task execution. However, the effective adjustment of Agentic during execution has not been well-studied.
  arXiv  Detail & Related papers  (2025-01-14T04:35:37Z)
• Opus: A Large Work Model for Complex Workflow Generation [0.0]
  Opus is a framework for generating and optimizing tasks tailored to complex Business Process Outsourcing (BPO) use cases. Our approach generates executables from Intention, defined as the alignment of Client Input, Client Output and Process Directed Context.
  arXiv  Detail & Related papers  (2024-11-30T20:00:41Z)
• WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models [105.46456444315693]
  We presentLLM, a data-centric framework to enhance the capability of large language models in workflow orchestration. It first constructs a large-scale fine-tuningBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. LlamaLlama demonstrates a strong capacity to orchestrate complex APIs, while also achieving notable generalization performance.
  arXiv  Detail & Related papers  (2024-11-08T09:58:02Z)
• AFlow: Automating Agentic Workflow Generation [36.61172223528231]
  Large language models (LLMs) have demonstrated remarkable potential in solving complex tasks across diverse domains. We introduce AFlow, an automated framework that efficiently explores this space using Monte Carlo Tree Search. Empirical evaluations across six benchmark datasets demonstrate AFlow's efficacy, yielding a 5.7% average improvement over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-10-14T17:40:40Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorFBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. We also present WorFEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms. We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation [87.39861573270173]
  We introduce the novel task of prompt-adaptive workflow generation, where the goal is to automatically tailor a workflow to each user prompt. We propose two LLM-based approaches to tackle this task: a tuning-based method that learns from user-preference data, and a training-free method that uses the LLM to select existing flows. Our work shows that prompt-dependent flow prediction offers a new pathway to improving text-to-image generation quality, complementing existing research directions in the field.
  arXiv  Detail & Related papers  (2024-10-02T16:43:24Z)
• Grammar-based evolutionary approach for automated workflow composition with domain-specific operators and ensemble diversity [0.36832029288386137]
  This paper introduces EvoFlow, a grammar-based evolutionary approach for automatic workflow composition (AWC) EvoFlow enhances the flexibility in designing workflow structures, empowering practitioners to select algorithms that best fit their specific requirements. Our findings show that EvoFlow's specialised genetic operators and updating mechanism substantially outperform current leading methods.
  arXiv  Detail & Related papers  (2024-02-03T11:29:14Z)
• Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelines [48.7576911714538]
  The proposed approach is aimed to automate the design of composite machine learning pipelines. It designs the pipelines with a customizable graph-based structure, analyzes the obtained results, and reproduces them. The software implementation on this approach is presented as an open-source framework.
  arXiv  Detail & Related papers  (2021-06-26T23:19:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.