LLMs in the Real World: Evaluating "AI" in Emergency Contexts

• URL: http://arxiv.org/abs/2607.00019v1
• Date: Fri, 29 May 2026 19:27:25 GMT
• Title: LLMs in the Real World: Evaluating "AI" in Emergency Contexts
• Abstract summary: We present a case study on the initial stages of an LLM-based machine translation application's deployment in a real-world context.<n>We identify a number of common misconceptions about technologies such as these, concluding with a set of concrete recommendations.
• Score: 2.115590859976976
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper offers a call to action. We urge our colleagues in the research community to play a greater role in the articulation of our findings to the public. To illustrate the stakes we present a case study on the initial stages of an LLM-based machine translation application's deployment in a real-world context: a text-2-911 system advertising capabilities in 55 languages for use in emergencies in which it may be difficult to call operators directly. We identify a number of common misconceptions about technologies such as these, concluding with a set of concrete recommendations and best practices for stakeholders at every stage of the development and deployment pipeline. While the advancement of scientific research often lies in solving the "hard" problems, we argue it is often the "easy" ones -- problems for which the latest technology is often unnecessary -- that are most overlooked.

Related papers

• What We Know about Responsible AI Practices in Industry: A Half Decade of Empirical Research [27.930796808132413]
  Responsible AI (RAI) has become a central concern for technology companies, regulators, and the public.<n>This work synthesizes current knowledge through a literature review of 161 empirical studies spanning six years.<n>Our synthesis reveals both meaningful progress and persistent challenges in industry RAI practice.
  arXiv  Detail & Related papers  (2026-08-11T03:29:48Z)
• LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026 [17.097147033209037]
  Late interaction retrieval methods, pioneered by ColBERT, have emerged as a powerful alternative to single-vector neural IR.<n>They have been demonstrated to deliver strong generalisation and robustness, particularly in out-of-domain settings.<n>They have recently been shown to be particularly well-suited for novel use cases, such as reasoning-based or cross-modality retrieval.
  arXiv  Detail & Related papers  (2025-11-01T08:21:33Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• Evaluating Large Language Models for Real-World Engineering Tasks [75.97299249823972]
  This paper introduces a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios.<n>Using this dataset, we evaluate four state-of-the-art Large Language Models (LLMs)<n>Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.
  arXiv  Detail & Related papers  (2025-05-12T14:05:23Z)
• Towards Automated Scoping of AI for Social Good Projects [8.600895107205083]
  We propose a Problem Scoping Agent (PSA) that generates comprehensive project proposals grounded in scientific literature and real-world knowledge.<n>We demonstrate that our PSA framework generates proposals comparable to those written by experts through a blind review and AI evaluations.
  arXiv  Detail & Related papers  (2025-04-28T17:29:51Z)
• Practical Considerations for Agentic LLM Systems [5.455744338342196]
  This paper frames actionable insights and considerations from the research community in the context of established application paradigms.<n> Namely, we position relevant research findings into four broad categories--Planning, Memory Tools, and Control Flow--based on common practices in application-focused literature.
  arXiv  Detail & Related papers  (2024-12-05T11:57:49Z)
• Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review [43.27239522837257]
  This paper is devoted to a comprehensive review of realizing the sample efficiency and generalization of RL algorithms through transfer and inverse reinforcement learning (T-IRL)<n>Our findings denote that a majority of recent research works have dealt with the aforementioned challenges by utilizing human-in-the-loop and sim-to-real strategies.<n>Under the IRL structure, training schemes that require a low number of experience transitions and extension of such frameworks to multi-agent and multi-intention problems have been the priority of researchers in recent years.
  arXiv  Detail & Related papers  (2024-11-15T15:18:57Z)
• A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges [27.681999552782372]
  Inverse Constrained Reinforcement Learning (ICRL) is the task of inferring the implicit constraints that expert agents adhere to, based on their demonstration data.<n>This article presents a categorical survey of the latest advances in ICRL.<n>It serves as a comprehensive reference for machine learning researchers and practitioners, as well as starters seeking to comprehend the definitions, advancements, and important challenges in ICRL.
  arXiv  Detail & Related papers  (2024-09-11T18:49:03Z)
• No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size [7.660415388174536]
  Large language models (LLMs) are playing a pivotal role in deploying strategic use cases across a range of organizations.<n>The issues and challenges involved in the successful utilization of LLMs can vary significantly depending on the size of the organization.
  arXiv  Detail & Related papers  (2024-07-21T16:11:00Z)
• LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application [54.984348122105516]
  Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework synergizes open-world knowledge with collaborative knowledge.<n>We propose an Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework that synergizes open-world knowledge with collaborative knowledge.
  arXiv  Detail & Related papers  (2024-05-07T04:00:30Z)
• A Systematic Literature Review on Large Language Models for Automated Program Repair [21.140070763968634]
  It is challenging for researchers to understand the current achievements, challenges, and potential opportunities.<n>This work provides the first systematic literature review to summarize the applications of Large Language Models in APR between 2020 and 2025.
  arXiv  Detail & Related papers  (2024-05-02T16:55:03Z)
• Large Multimodal Agents: A Survey [78.81459893884737]
  Large language models (LLMs) have achieved superior performance in powering text-based AI agents. There is an emerging research trend focused on extending these LLM-powered AI agents into the multimodal domain. This review aims to provide valuable insights and guidelines for future research in this rapidly evolving field.
  arXiv  Detail & Related papers  (2024-02-23T06:04:23Z)
• Responsible AI Considerations in Text Summarization Research: A Review of Current Practices [89.85174013619883]
  We focus on text summarization, a common NLP task largely overlooked by the responsible AI community. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020-2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
  arXiv  Detail & Related papers  (2023-11-18T15:35:36Z)
• Re-Reading Improves Reasoning in Large Language Models [87.46256176508376]
  We introduce a simple, yet general and effective prompting method, Re2, to enhance the reasoning capabilities of off-the-shelf Large Language Models (LLMs) Unlike most thought-eliciting prompting methods, such as Chain-of-Thought (CoT), Re2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process. We evaluate Re2 on extensive reasoning benchmarks across 14 datasets, spanning 112 experiments, to validate its effectiveness and generality.
  arXiv  Detail & Related papers  (2023-09-12T14:36:23Z)
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey [53.73359052511171]
  Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. We present a framework for curriculum learning (CL) in RL, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals.
  arXiv  Detail & Related papers  (2020-03-10T20:41:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.