MAFA: A Multi-Agent Framework for Enterprise-Scale Annotation with Configurable Task Adaptation

• URL: http://arxiv.org/abs/2510.14184v1
• Date: Thu, 16 Oct 2025 00:30:08 GMT
• Title: MAFA: A Multi-Agent Framework for Enterprise-Scale Annotation with Configurable Task Adaptation
• Abstract summary: We present a production-deployed system that transforms enterprise-scale annotation through multi-agent collaboration.<n>Mafa has eliminated a 1 million utterance backlog while achieving, on average, 86% agreement with human annotators.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present MAFA (Multi-Agent Framework for Annotation), a production-deployed system that transforms enterprise-scale annotation workflows through configurable multi-agent collaboration. Addressing the critical challenge of annotation backlogs in financial services, where millions of customer utterances require accurate categorization, MAFA combines specialized agents with structured reasoning and a judge-based consensus mechanism. Our framework uniquely supports dynamic task adaptation, allowing organizations to define custom annotation types (FAQs, intents, entities, or domain-specific categories) through configuration rather than code changes. Deployed at JP Morgan Chase, MAFA has eliminated a 1 million utterance backlog while achieving, on average, 86% agreement with human annotators, annually saving over 5,000 hours of manual annotation work. The system processes utterances with annotation confidence classifications, which are typically 85% high, 10% medium, and 5% low across all datasets we tested. This enables human annotators to focus exclusively on ambiguous and low-coverage cases. We demonstrate MAFA's effectiveness across multiple datasets and languages, showing consistent improvements over traditional and single-agent annotation baselines: 13.8% higher Top-1 accuracy, 15.1% improvement in Top-5 accuracy, and 16.9% better F1 in our internal intent classification dataset and similar gains on public benchmarks. This work bridges the gap between theoretical multi-agent systems and practical enterprise deployment, providing a blueprint for organizations facing similar annotation challenges.

Related papers

• AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility [104.46861849039357]
  Agent systems are advancing quickly across domains, but their evaluation remains fragmented.<n>The root problem is the lack of an open, agent-agnostic assessment interface.<n>We advocate Agentified Agent Assessment (AAA), where evaluation is performed by judge agents and all participants interact through standardized protocols.
  arXiv  Detail & Related papers  (2026-06-11T17:23:54Z)
• SAGE: A Service Agent Graph-guided Evaluation Benchmark [27.342044311161654]
  We propose SAGE (Service Agent Graph-guided Evaluation), a universal benchmark for automated, dual-axis assessment.<n>SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance.<n>We also observe Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures.
  arXiv  Detail & Related papers  (2026-04-10T12:55:23Z)
• OFA-MAS: One-for-All Multi-Agent System Topology Design based on Mixture-of-Experts Graph Generative Models [57.94189874119267]
  Multi-Agent Systems (MAS) offer a powerful paradigm for solving complex problems.<n>Current graph learning-based design methodologies often adhere to a "one-for-one" paradigm.<n>We propose OFA-TAD, a one-for-all framework that generates adaptive collaboration graphs for any task described in natural language.
  arXiv  Detail & Related papers  (2026-01-19T12:23:44Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• Alita-G: Self-Evolving Generative Agent for Agent Generation [54.49365835457433]
  We present ALITA-G, a framework that transforms a general-purpose agent into a domain expert.<n>In this framework, a generalist agent executes a curated suite of target-domain tasks.<n>It attains strong gains while reducing computation costs.
  arXiv  Detail & Related papers  (2025-10-27T17:59:14Z)
• How can we assess human-agent interactions? Case studies in software agent design [52.953425368394306]
  We make two major steps towards the rigorous assessment of human-agent interactions.<n>We propose PULSE, a framework for more efficient human-centric evaluation of agent designs.<n>We deploy the framework on a large-scale web platform built around the open-source software agent OpenHands.
  arXiv  Detail & Related papers  (2025-10-10T19:04:28Z)
• AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition [27.312190686305588]
  Large language models (LLMs) have shown strong potential in automated scoring.<n>Their use as end-to-end raters faces challenges such as low accuracy, prompt sensitivity, limited interpretability, and rubric misalignment.<n>We propose AutoSCORE, a multi-agent LLM framework enhancing automated scoring via rubric-aligned Structured COmponent REcognition.
  arXiv  Detail & Related papers  (2025-09-26T05:45:14Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• HiMATE: A Hierarchical Multi-Agent Framework for Machine Translation Evaluation [39.7293877954587]
  HiMATE is a Hierarchical Multi-Agent Framework for Machine Translation Evaluation.<n>We develop a hierarchical multi-agent system grounded in the MQM error typology, enabling granular evaluation of subtype errors.<n> Empirically, HiMATE outperforms competitive baselines across different datasets in conducting human-aligned evaluations.
  arXiv  Detail & Related papers  (2025-05-22T06:24:08Z)
• MAFA: A multi-agent framework for annotation [0.0]
  We introduce a multi-agent framework for annotation that combines specialized agents with different approaches and a judge agent that reranks candidates to produce optimal results.<n>Our framework is particularly effective at handling ambiguous queries, making it well-suited for deployment in production banking applications.
  arXiv  Detail & Related papers  (2025-05-19T19:16:37Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)
• A Unified and Efficient Coordinating Framework for Autonomous DBMS Tuning [34.85351481228439]
  We propose a unified coordinating framework to efficiently utilize existing ML-based agents. We show that it can effectively utilize different ML-based agents and find better configurations with 1.414.1X speedups on the workload execution time.
  arXiv  Detail & Related papers  (2023-03-10T05:27:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.