LinguistAgent: A Reflective Multi-Model Platform for Automated Linguistic Annotation

• URL: http://arxiv.org/abs/2602.05493v1
• Date: Thu, 05 Feb 2026 09:55:19 GMT
• Title: LinguistAgent: A Reflective Multi-Model Platform for Automated Linguistic Annotation
• Abstract summary: This paper introduces LinguistAgent, an integrated platform that leverages a reflective multi-model architecture to automate linguistic annotation.<n>The system implements a dual-agent workflow, comprising an Annotator and a Reviewer, to simulate a professional peer-review process.<n>We demonstrate LinguistAgent's efficacy using the task of metaphor identification as an example, providing real-time token-level evaluation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Data annotation remains a significant bottleneck in the Humanities and Social Sciences, particularly for complex semantic tasks such as metaphor identification. While Large Language Models (LLMs) show promise, a significant gap remains between the theoretical capability of LLMs and their practical utility for researchers. This paper introduces LinguistAgent, an integrated, user-friendly platform that leverages a reflective multi-model architecture to automate linguistic annotation. The system implements a dual-agent workflow, comprising an Annotator and a Reviewer, to simulate a professional peer-review process. LinguistAgent supports comparative experiments across three paradigms: Prompt Engineering (Zero/Few-shot), Retrieval-Augmented Generation, and Fine-tuning. We demonstrate LinguistAgent's efficacy using the task of metaphor identification as an example, providing real-time token-level evaluation (Precision, Recall, and $F_1$ score) against human gold standards. The application and codes are released on https://github.com/Bingru-Li/LinguistAgent.

Related papers

• Automated Interpretability and Feature Discovery in Language Models with Agents [3.660409142843126]
  We introduce an autonomous multiagent framework for mechanistic interpretability that automates both explaining and finding internal features in large language models.<n>We show that agent-driven empirical loops yield sharper and more falsifiable explanations than one-shot labels.
  arXiv  Detail & Related papers  (2026-05-02T17:53:30Z)
• SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation [20.300369465395786]
  SignAgent is a novel agentic framework for scalable, linguistically-grounded Sign Language (SL) annotation and dataset curation.<n>Our results demonstrate that our agentic approach achieves strong performance for large-scale, linguistically-aware data annotation and curation.
  arXiv  Detail & Related papers  (2026-03-19T15:52:49Z)
• AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent [57.10083973844841]
  AgentArk is a novel framework to distill multi-agent dynamics into the weights of a single model.<n>We investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios.<n>By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents.
  arXiv  Detail & Related papers  (2026-02-03T19:18:28Z)
• Bridging the Knowledge Void: Inference-time Acquisition of Unfamiliar Programming Languages for Coding Tasks [22.908904483320953]
  Large Language Models (LLMs) in coding tasks are often a reflection of their extensive pre-training corpora.<n>We propose ILA-agent, a general ILA framework that equips LLMs with a set of behavioral primitives.<n>We instantiate ILA-agent for Cangjie and evaluate its performance across code generation, translation, and program repair tasks.
  arXiv  Detail & Related papers  (2026-01-16T09:06:47Z)
• Code-in-the-Loop Forensics: Agentic Tool Use for Image Forgery Detection [59.04089915447622]
  ForenAgent is an interactive IFD framework that enables MLLMs to autonomously generate, execute, and refine Python-based low-level tools around the detection objective.<n>Inspired by human reasoning, we design a dynamic reasoning loop comprising global perception, local focusing, iterative probing, and holistic adjudication.<n>Experiments show that ForenAgent exhibits emergent tool-use competence and reflective reasoning on challenging IFD tasks.
  arXiv  Detail & Related papers  (2025-12-18T08:38:44Z)
• Evaluating Large Language Models as Expert Annotators [17.06186816803593]
  This paper investigates whether top-performing language models can serve as direct alternatives to human expert annotators.<n>We evaluate both individual LLMs and multi-agent approaches across three highly specialized domains: finance, biomedicine, and law.<n>Our empirical results reveal that individual LLMs equipped with inference-time techniques show only marginal or even negative performance gains.
  arXiv  Detail & Related papers  (2025-08-11T10:19:10Z)
• Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Survey [52.00674453604779]
  This survey organizes the literature on the design and optimization of emerging structures around post-trained LMs.<n>We refer to this overarching structure as scaffolded LMs and focus on LMs that are integrated into multi-step processes with tools.
  arXiv  Detail & Related papers  (2024-10-21T18:06:25Z)
• Symbolic Learning Enables Self-Evolving Agents [55.625275970720374]
  We introduce agent symbolic learning, a systematic framework that enables language agents to optimize themselves on their own. Agent symbolic learning is designed to optimize the symbolic network within language agents by mimicking two fundamental algorithms in connectionist learning. We conduct proof-of-concept experiments on both standard benchmarks and complex real-world tasks.
  arXiv  Detail & Related papers  (2024-06-26T17:59:18Z)
• An Empirical Evaluation of Pre-trained Large Language Models for Repairing Declarative Formal Specifications [7.286515881369693]
  This paper systematically investigates the capacity of Large Language Models (LLMs) to repair declarative specifications in Alloy.<n>We designed 12 different repair settings, encompassing single-agent and dual-agent paradigms, utilizing various LLMs.<n>Our study reveals that dual-agent with auto-prompting setup outperforms the other settings, albeit with a marginal increase in the number of iterations and token usage.
  arXiv  Detail & Related papers  (2024-04-17T03:46:38Z)
• Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization [103.70896967077294]
  This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model. Our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model. Experimental results on various tasks demonstrate that the language agents improve over time.
  arXiv  Detail & Related papers  (2023-08-04T06:14:23Z)
• A Hierarchical Model for Spoken Language Recognition [29.948719321162883]
  Spoken language recognition ( SLR) refers to the automatic process used to determine the language present in a speech sample. We propose a novel hierarchical approach were two PLDA models are trained, one to generate scores for clusters of highly related languages and a second one to generate scores conditional to each cluster. We show that this hierarchical approach consistently outperforms the non-hierarchical one for detection of highly related languages.
  arXiv  Detail & Related papers  (2022-01-04T22:10:36Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.