Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows

• URL: http://arxiv.org/abs/2312.11681v3
• Date: Mon, 6 May 2024 18:18:46 GMT
• Title: Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows
• Authors: Madeleine Grunde-McLaughlin, Michelle S. Lam, Ranjay Krishna, Daniel S. Weld, Jeffrey Heer,
• Abstract summary: LLM chains enable complex tasks by decomposing work into a sequence of subtasks. Crowdsourcing addresses errors analogous to the way crowdsourcing address human error.
• Score: 37.60760400107501
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM chains enable complex tasks by decomposing work into a sequence of subtasks. Similarly, the more established techniques of crowdsourcing workflows decompose complex tasks into smaller tasks for human crowdworkers. Chains address LLM errors analogously to the way crowdsourcing workflows address human error. To characterize opportunities for LLM chaining, we survey 107 papers across the crowdsourcing and chaining literature to construct a design space for chain development. The design space covers a designer's objectives and the tactics used to build workflows. We then surface strategies that mediate how workflows use tactics to achieve objectives. To explore how techniques from crowdsourcing may apply to chaining, we adapt crowdsourcing workflows to implement LLM chains across three case studies: creating a taxonomy, shortening text, and writing a short story. From the design space and our case studies, we identify takeaways for effective chain design and raise implications for future research and development.

Related papers

• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorFBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. We also present WorFEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms. We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation [87.39861573270173]
  We introduce the novel task of prompt-adaptive workflow generation, where the goal is to automatically tailor a workflow to each user prompt. We propose two LLM-based approaches to tackle this task: a tuning-based method that learns from user-preference data, and a training-free method that uses the LLM to select existing flows. Our work shows that prompt-dependent flow prediction offers a new pathway to improving text-to-image generation quality, complementing existing research directions in the field.
  arXiv  Detail & Related papers  (2024-10-02T16:43:24Z)
• Optimizing Token Usage on Large Language Model Conversations Using the Design Structure Matrix [49.1574468325115]
  Large Language Models become ubiquitous in many sectors and tasks. There is a need to reduce token usage, overcoming challenges such as short context windows, limited output sizes, and costs associated with token intake and generation. This work brings the Design Structure Matrix from the engineering design discipline into LLM conversation optimization.
  arXiv  Detail & Related papers  (2024-10-01T14:38:36Z)
• From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models [56.9134620424985]
  Cross-modal reasoning (CMR) is increasingly recognized as a crucial capability in the progression toward more sophisticated artificial intelligence systems. The recent trend of deploying Large Language Models (LLMs) to tackle CMR tasks has marked a new mainstream of approaches for enhancing their effectiveness. This survey offers a nuanced exposition of current methodologies applied in CMR using LLMs, classifying these into a detailed three-tiered taxonomy.
  arXiv  Detail & Related papers  (2024-09-19T02:51:54Z)
• RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Science [0.46560775769914914]
  Large language models (LLMs) have enhanced our ability to rapidly analyze and classify unstructured natural language data. However, concerns regarding cost, network limitations, and security constraints have posed challenges for their integration into work processes. In this study, we adopt a systems design approach to employing LLMs as imperfect data annotators for downstream supervised learning tasks.
  arXiv  Detail & Related papers  (2024-08-15T15:28:37Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability [45.235173351109374]
  Large Language Models (LLMs) have the potential to revolutionize automated traceability. This paper explores the process of prompt engineering to extract link predictions from an LLM.
  arXiv  Detail & Related papers  (2023-08-01T01:56:22Z)
• Power-up! What Can Generative Models Do for Human Computation Workflows? [13.484359389266864]
  Investigation into large language models (LLMs) as part of crowdsourcing remains an under-explored space. From an empirical standpoint, little is currently understood about how LLMs can improve the effectiveness of crowdsourcing.
  arXiv  Detail & Related papers  (2023-07-05T12:35:29Z)
• AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts [12.73129785710807]
  We introduce the concept of Chaining LLM steps together, where the output of one step becomes the input for the next, thus aggregating the gains per step. In a 20-person user study, we found that Chaining not only improved the quality of task outcomes, but also significantly enhanced system transparency, controllability, and sense of collaboration.
  arXiv  Detail & Related papers  (2021-10-04T19:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.