Rethinking the Evaluating Framework for Natural Language Understanding in AI Systems: Language Acquisition as a Core for Future Metrics

• URL: http://arxiv.org/abs/2309.11981v3
• Date: Thu, 5 Oct 2023 02:58:52 GMT
• Title: Rethinking the Evaluating Framework for Natural Language Understanding in AI Systems: Language Acquisition as a Core for Future Metrics
• Authors: Patricio Vera, Pedro Moya and Lisa Barraza
• Abstract summary: In the burgeoning field of artificial intelligence (AI), the unprecedented progress of large language models (LLMs) in natural language processing (NLP) offers an opportunity to revisit the entire approach of traditional metrics of machine intelligence. Our paper proposes a paradigm shift from the established Turing Test towards an all-embracing framework that hinges on language acquisition.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the burgeoning field of artificial intelligence (AI), the unprecedented progress of large language models (LLMs) in natural language processing (NLP) offers an opportunity to revisit the entire approach of traditional metrics of machine intelligence, both in form and content. As the realm of machine cognitive evaluation has already reached Imitation, the next step is an efficient Language Acquisition and Understanding. Our paper proposes a paradigm shift from the established Turing Test towards an all-embracing framework that hinges on language acquisition, taking inspiration from the recent advancements in LLMs. The present contribution is deeply tributary of the excellent work from various disciplines, point out the need to keep interdisciplinary bridges open, and delineates a more robust and sustainable approach.

Related papers

• MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting [53.77590764277568]
  We introduce a novel MoE-CT architecture that separates the base model's learning from the multilingual expansion process. Our design freezes the original LLM parameters, thus safeguarding its performance in high-resource languages, while an appended MoE module, trained on diverse language datasets, augments low-resource language proficiency.
  arXiv  Detail & Related papers  (2024-06-25T11:03:45Z)
• LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments [70.91258869156353]
  We introduce LangSuitE, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuitE offers adaptability to diverse environments without multiple simulation engines. We devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information.
  arXiv  Detail & Related papers  (2024-06-24T03:36:29Z)
• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• Deep Learning Approaches for Improving Question Answering Systems in Hepatocellular Carcinoma Research [0.0]
  In recent years, advancements in natural language processing (NLP) have been fueled by deep learning techniques. BERT and GPT-3, trained on vast amounts of data, have revolutionized language understanding and generation. This paper delves into the current landscape and future prospects of large-scale model-based NLP.
  arXiv  Detail & Related papers  (2024-02-25T09:32:17Z)
• LB-KBQA: Large-language-model and BERT based Knowledge-Based Question and Answering System [7.626368876843794]
  We propose a novel KBQA system based on a Large Language Model(LLM) and BERT (LB-KBQA) With the help of generative AI, our proposed method could detect newly appeared intent and acquire new knowledge. In experiments on financial domain question answering, our model has demonstrated superior effectiveness.
  arXiv  Detail & Related papers  (2024-02-05T16:47:17Z)
• Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents [80.5213198675411]
  Large language models (LLMs) have dramatically enhanced the field of language intelligence. LLMs leverage the intriguing chain-of-thought (CoT) reasoning techniques, obliging them to formulate intermediate steps en route to deriving an answer. Recent research endeavors have extended CoT reasoning methodologies to nurture the development of autonomous language agents.
  arXiv  Detail & Related papers  (2023-11-20T14:30:55Z)
• Building Trust in Conversational AI: A Comprehensive Review and Solution Architecture for Explainable, Privacy-Aware Systems using LLMs and Knowledge Graph [0.33554367023486936]
  We introduce a comprehensive tool that provides an in-depth review of over 150 Large Language Models (LLMs) Building on this foundation, we propose a novel functional architecture that seamlessly integrates the structured dynamics of Knowledge Graphs with the linguistic capabilities of LLMs. Our architecture adeptly blends linguistic sophistication with factual rigour and further strengthens data security through Role-Based Access Control.
  arXiv  Detail & Related papers  (2023-08-13T22:47:51Z)
• From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought [124.40905824051079]
  We propose rational meaning construction, a computational framework for language-informed thinking. We frame linguistic meaning as a context-sensitive mapping from natural language into a probabilistic language of thought. We show that LLMs can generate context-sensitive translations that capture pragmatically-appropriate linguistic meanings. We extend our framework to integrate cognitively-motivated symbolic modules.
  arXiv  Detail & Related papers  (2023-06-22T05:14:00Z)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate [95.10641301155232]
  We present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.
  arXiv  Detail & Related papers  (2023-05-23T17:55:11Z)
• Knowledge Engineering in the Long Game of Artificial Intelligence: The Case of Speech Acts [0.6445605125467572]
  This paper describes principles and practices of knowledge engineering that enable the development of holistic language-endowed intelligent agents. We focus on dialog act modeling, a task that has been widely pursued in linguistics, cognitive modeling, and statistical natural language processing.
  arXiv  Detail & Related papers  (2022-02-02T14:05:12Z)
• Language Generation for Broad-Coverage, Explainable Cognitive Systems [0.0]
  This paper describes recent progress on natural language generation for language-endowed intelligent agents (LEIAs) developed within the OntoAgent cognitive architecture. It uses the same knowledge bases, theory of computational linguistics, agent architecture, and methodology of developing broad-coverage capabilities over time while still supporting near-term applications.
  arXiv  Detail & Related papers  (2022-01-25T16:09:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.