MEGAnno+: A Human-LLM Collaborative Annotation System
- URL: http://arxiv.org/abs/2402.18050v1
- Date: Wed, 28 Feb 2024 04:58:07 GMT
- Title: MEGAnno+: A Human-LLM Collaborative Annotation System
- Authors: Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, Dan Zhang
- Abstract summary: Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks.
Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context.
We advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels.
- Score: 6.10245234757137
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Large language models (LLMs) can label data faster and cheaper than humans
for various NLP tasks. Despite their prowess, LLMs may fall short in
understanding of complex, sociocultural, or domain-specific context,
potentially leading to incorrect annotations. Therefore, we advocate a
collaborative approach where humans and LLMs work together to produce reliable
and high-quality labels. We present MEGAnno+, a human-LLM collaborative
annotation system that offers effective LLM agent and annotation management,
convenient and robust LLM annotation, and exploratory verification of LLM
labels by humans.
Related papers
- Large language models enabled multiagent ensemble method for efficient EHR data labeling [9.481473827205159]
This study introduces a novel multiagent ensemble method powered by LLMs to address a key challenge in ML - data labeling.
By using the ensemble LLMs and natural language processing, we labeled MIMIC-IV ECG dataset of 623,566 ECG reports with an estimated accuracy of 98.2%.
We applied the ensemble LLMs method to identify SDOH from social history sections of 1,405 EHR clinical notes, also achieving competitive performance.
arXiv Detail & Related papers (2024-10-21T22:12:00Z) - The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs [42.72336063802124]
Multi-modal Large Language Models (MLLMs) have exhibited impressive capability.
Many deficiencies of MLLMs have been found compared to human intelligence, $textite.g.$, hallucination.
We propose benchmarking an essential but usually overlooked intelligence: $textbfassociation$, a human's basic capability to link observation and prior practice memory.
arXiv Detail & Related papers (2024-10-02T10:58:54Z) - CoRA: Collaborative Information Perception by Large Language Model's Weights for Recommendation [13.867950651601483]
Involving collaborative information in Large Language Models (LLMs) is a promising technique for adapting LLMs for recommendation.
Existing methods achieve this by concatenating collaborative features with text tokens into a unified sequence input.
We propose a new paradigm, textbfCollaborative textbfLoRA, with a collaborative query generator.
arXiv Detail & Related papers (2024-08-20T08:36:59Z) - Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing [56.75702900542643]
We introduce AlphaLLM for the self-improvements of Large Language Models.
It integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving loop.
Our experimental results show that AlphaLLM significantly enhances the performance of LLMs without additional annotations.
arXiv Detail & Related papers (2024-04-18T15:21:34Z) - Knowledge Fusion of Large Language Models [73.28202188100646]
This paper introduces the notion of knowledge fusion for large language models (LLMs)
We externalize their collective knowledge and unique strengths, thereby elevating the capabilities of the target model beyond those of any individual source LLM.
Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation.
arXiv Detail & Related papers (2024-01-19T05:02:46Z) - Boosting Large Language Model for Speech Synthesis: An Empirical Study [86.89548753080432]
Large language models (LLMs) have made significant advancements in natural language processing and are concurrently extending the language ability to other modalities, such as speech and vision.
We conduct a comprehensive empirical exploration of boosting LLMs with the ability to generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech synthesis model VALL-E.
We compare three integration methods between LLMs and speech models, including directly fine-tuned LLMs, superposed layers of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text encoder
arXiv Detail & Related papers (2023-12-30T14:20:04Z) - CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large
Language Models for Data Annotation [94.59630161324013]
We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale.
Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
arXiv Detail & Related papers (2023-10-24T08:56:49Z) - Label Supervised LLaMA Finetuning [13.939718306233617]
In this paper, we introduce a label-supervised adaptation for Large Language Models (LLMs)
We extract latent representations from the final LLaMA layer and project them into the label space to compute the cross-entropy loss.
Remarkably, without intricate prompt engineering or external knowledge, LS-LLaMA substantially outperforms LLMs ten times its size in scale.
arXiv Detail & Related papers (2023-10-02T13:53:03Z) - Low-code LLM: Graphical User Interface over Large Language Models [115.08718239772107]
This paper introduces a novel human-LLM interaction framework, Low-code LLM.
It incorporates six types of simple low-code visual programming interactions to achieve more controllable and stable responses.
We highlight three advantages of the low-code LLM: user-friendly interaction, controllable generation, and wide applicability.
arXiv Detail & Related papers (2023-04-17T09:27:40Z) - Can Large Language Models Transform Computational Social Science? [79.62471267510963]
Large Language Models (LLMs) are capable of performing many language processing tasks zero-shot (without training data)
This work provides a road map for using LLMs as Computational Social Science tools.
arXiv Detail & Related papers (2023-04-12T17:33:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.