PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

• URL: http://arxiv.org/abs/2607.06008v2
• Date: Thu, 09 Jul 2026 11:01:43 GMT
• Title: PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents
• Abstract summary: Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments.<n>Most existing benchmarks implicitly assume a monolingual setting, where the entire execution process is conducted within a single language.<n>We introduce PolyWorkBench, a benchmark for evaluating agents on multilingual long-horizon workplace work.
• Score: 72.96251271825038
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments. However, most existing benchmarks implicitly assume a monolingual setting, where the entire execution process, including reasoning, tool invocation, and output generation, is conducted within a single language. In contrast, real-world applications often involve multilingual inputs and outputs within a unified workflow, yet the interaction between multilinguality and agentic execution remains underexplored. In this work, we introduce PolyWorkBench, a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows. PolyWorkBench consists of 67 tasks across five domains, including commerce, knowledge work, legal analysis, localization, and manufacturing, where agents must process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. To enable comprehensive evaluation, we propose a hybrid framework that combines structural grading, executable verification, and LLM-based semantic assessment. This design allows us to capture both functional correctness and linguistic consistency across complex workflows. Empirical results show that state-of-the-art LLM agents suffer significant performance degradation in multilingual workflow settings compared to monolingual counterparts. Our analysis suggests that multilinguality introduces compounding effects across reasoning and execution steps, highlighting the importance of jointly modeling language variation and procedural decision-making in agent evaluation.

Related papers

• WorldBench: Culturally Grounded Benchmark for Multilingual Agents [69.76002914143531]
  Existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios.<n>We present WorldBench: a comprehensive, multilingual benchmark, where agents can act in a sandbox via structured actions.<n>WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators.
  arXiv  Detail & Related papers  (2026-09-01T10:53:07Z)
• Bridging the Knowledge Void: Inference-time Acquisition of Unfamiliar Programming Languages for Coding Tasks [22.908904483320953]
  Large Language Models (LLMs) in coding tasks are often a reflection of their extensive pre-training corpora.<n>We propose ILA-agent, a general ILA framework that equips LLMs with a set of behavioral primitives.<n>We instantiate ILA-agent for Cangjie and evaluate its performance across code generation, translation, and program repair tasks.
  arXiv  Detail & Related papers  (2026-01-16T09:06:47Z)
• From over-reliance to smart integration: using Large-Language Models as translators between specialized modeling and simulation tools [0.3749861135832073]
  Large Language Models (LLMs) offer transformative potential for Modeling & Simulation (M&S)<n>Over-reliance risks compromising quality due to ambiguities, logical shortcuts, and hallucinations.<n>This paper advocates integrating LLMs as translators between specialized tools to mitigate complexity in M&S tasks.
  arXiv  Detail & Related papers  (2025-06-11T02:39:08Z)
• EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models [64.70546873396624]
  We present the Extremely Complex Instruction Following Benchmark (EIFBENCH) for evaluating large language models (LLMs)<n>EIFBENCH includes multi-task scenarios that enable comprehensive assessment across diverse task types concurrently.<n>We also propose the Segment Policy Optimization (SegPO) algorithm to enhance the LLM's ability to accurately fulfill multi-task workflow.
  arXiv  Detail & Related papers  (2025-06-10T02:39:55Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• Collaboration is all you need: LLM Assisted Safe Code Translation [4.3764649156831235]
  UniTranslator is a framework that re-imagines code translation as a collaborative endeavor among multiple, compact LLMs.<n>By orchestrating the interaction of specialized agents, UniTranslator achieves a level of accuracy and efficiency that rivals larger, monolithic models.
  arXiv  Detail & Related papers  (2025-03-14T09:42:07Z)
• LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models [89.13128402847943]
  We present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision.<n>LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks.<n>We introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages.
  arXiv  Detail & Related papers  (2025-01-01T15:43:07Z)
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs [84.24644520272835]
  We introduce P-MMEval, a large-scale benchmark covering effective fundamental and capability-specialized datasets.<n>P-MMEval delivers consistent language coverage across various datasets and provides parallel samples.<n>We conduct extensive experiments on representative multilingual model series to compare performances across models and tasks.
  arXiv  Detail & Related papers  (2024-11-14T01:29:36Z)
• Multilingual Large Language Models Are Not (Yet) Code-Switchers [41.47534626749588]
  Large Language Models (LLMs) have recently shown great capabilities in a wide range of tasks. The practice of alternating languages within an utterance remains relatively uncharted. We argue that current "multilingualism" in LLMs does not inherently imply proficiency with code-switching texts.
  arXiv  Detail & Related papers  (2023-05-23T16:50:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.