Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows
- URL: http://arxiv.org/abs/2312.11681v3
- Date: Mon, 6 May 2024 18:18:46 GMT
- Title: Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows
- Authors: Madeleine Grunde-McLaughlin, Michelle S. Lam, Ranjay Krishna, Daniel S. Weld, Jeffrey Heer,
- Abstract summary: LLM chains enable complex tasks by decomposing work into a sequence of subtasks.
Crowdsourcing addresses errors analogous to the way crowdsourcing address human error.
- Score: 37.60760400107501
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: LLM chains enable complex tasks by decomposing work into a sequence of subtasks. Similarly, the more established techniques of crowdsourcing workflows decompose complex tasks into smaller tasks for human crowdworkers. Chains address LLM errors analogously to the way crowdsourcing workflows address human error. To characterize opportunities for LLM chaining, we survey 107 papers across the crowdsourcing and chaining literature to construct a design space for chain development. The design space covers a designer's objectives and the tactics used to build workflows. We then surface strategies that mediate how workflows use tactics to achieve objectives. To explore how techniques from crowdsourcing may apply to chaining, we adapt crowdsourcing workflows to implement LLM chains across three case studies: creating a taxonomy, shortening text, and writing a short story. From the design space and our case studies, we identify takeaways for effective chain design and raise implications for future research and development.
Related papers
- Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases.
We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning.
Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
arXiv Detail & Related papers (2024-06-19T00:28:58Z) - Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks [22.66167973623777]
Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning (ICL) capabilities.
This paper investigates whether LLMs can generalize from labeled examples of predefined tasks to novel tasks.
We show that cross-task prompting leads to a remarkable performance boost of 107% for LLaMA-2 7B, 18.6% for LLaMA-2 13B, and 3.2% for GPT 3.5 on average over zero-shot prompting.
arXiv Detail & Related papers (2024-05-17T05:20:49Z) - Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
Large Language Models (LLMs) have presented impressive performance across several transformative tasks.
However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs.
We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
arXiv Detail & Related papers (2024-03-12T13:31:14Z) - Instruction Tuning for Large Language Models: A Survey [52.86322823501338]
We make a systematic review of the literature, including the general methodology of IT, the construction of IT datasets, the training of IT models, and applications to different modalities, domains and applications.
We also review the potential pitfalls of IT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies and suggest some avenues for fruitful research.
arXiv Detail & Related papers (2023-08-21T15:35:16Z) - Revisiting Prompt Engineering via Declarative Crowdsourcing [16.624577543520093]
Large language models (LLMs) are incredibly powerful at comprehending and generating data in the form of text, but are brittle and error-prone.
We put forth a vision for declarative prompt engineering.
Preliminary case studies on sorting, entity resolution, and imputation demonstrate the promise of our approach.
arXiv Detail & Related papers (2023-08-07T18:04:12Z) - Prompts Matter: Insights and Strategies for Prompt Engineering in
Automated Software Traceability [45.235173351109374]
Large Language Models (LLMs) have the potential to revolutionize automated traceability.
This paper explores the process of prompt engineering to extract link predictions from an LLM.
arXiv Detail & Related papers (2023-08-01T01:56:22Z) - Power-up! What Can Generative Models Do for Human Computation Workflows? [13.484359389266864]
Investigation into large language models (LLMs) as part of crowdsourcing remains an under-explored space.
From an empirical standpoint, little is currently understood about how LLMs can improve the effectiveness of crowdsourcing.
arXiv Detail & Related papers (2023-07-05T12:35:29Z) - OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs.
Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
arXiv Detail & Related papers (2023-05-24T10:08:04Z) - Low-code LLM: Graphical User Interface over Large Language Models [115.08718239772107]
This paper introduces a novel human-LLM interaction framework, Low-code LLM.
It incorporates six types of simple low-code visual programming interactions to achieve more controllable and stable responses.
We highlight three advantages of the low-code LLM: user-friendly interaction, controllable generation, and wide applicability.
arXiv Detail & Related papers (2023-04-17T09:27:40Z) - AI Chains: Transparent and Controllable Human-AI Interaction by Chaining
Large Language Model Prompts [12.73129785710807]
We introduce the concept of Chaining LLM steps together, where the output of one step becomes the input for the next, thus aggregating the gains per step.
In a 20-person user study, we found that Chaining not only improved the quality of task outcomes, but also significantly enhanced system transparency, controllability, and sense of collaboration.
arXiv Detail & Related papers (2021-10-04T19:59:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.