Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles

• URL: http://arxiv.org/abs/2502.00817v1
• Date: Sun, 02 Feb 2025 14:53:14 GMT
• Title: Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
• Authors: Zheng-Lin Lin, Yu-Fei Shih, Shu-Kai Hsieh,
• Abstract summary: This paper investigates the utilization of Large Language Models (LLMs) for solving complex linguistic puzzles.<n>Using datasets from the Puzzling Machine Competition and various Linguistics Olympiads, we employ a comprehensive set of metrics to assess the performance of GPT-4 0603.
• Score: 0.6144680854063939
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper investigates the utilization of Large Language Models (LLMs) for solving complex linguistic puzzles, a domain requiring advanced reasoning and adept translation capabilities akin to human cognitive processes. We explore specific prompting techniques designed to enhance ability of LLMs to reason and elucidate their decision-making pathways, with a focus on Input-Output Prompting (IO), Chain-of-Thought Prompting (CoT), and Solo Performance Prompting (SPP). Utilizing datasets from the Puzzling Machine Competition and various Linguistics Olympiads, we employ a comprehensive set of metrics to assess the performance of GPT-4 0603, a prominent LLM, across these prompting methods. Our findings illuminate the potential of LLMs in linguistic reasoning and complex translation tasks, highlighting their capabilities and identifying limitations in the context of linguistic puzzles. This research contributes significantly to the broader field of Natural Language Processing (NLP) by providing insights into the optimization of LLM applications for improved reasoning and translation accuracy, thereby enriching the ongoing dialogue in NLP advancements.

Related papers

• Linguistic Blind Spots of Large Language Models [14.755831733659699]
  We study the performance of recent large language models (LLMs) on linguistic annotation tasks. We find that recent LLMs show limited efficacy in addressing linguistic queries and often struggle with linguistically complex inputs. Our results provide insights to inform future advancements in LLM design and development.
  arXiv  Detail & Related papers  (2025-03-25T01:47:13Z)
• Explicit Learning and the LLM in Machine Translation [20.630120942837564]
  This study explores the capacity of large language models (LLMs) for explicit learning. Using constructed languages generated by means as controlled test environments, we designed experiments to assess an LLM's ability to explicitly learn and apply grammar rules. Supervised fine-tuning on chains of thought significantly enhances LLM performance but struggles to generalize to typologically novel or more complex linguistic features.
  arXiv  Detail & Related papers  (2025-03-12T14:57:08Z)
• IOLBENCH: Benchmarking LLMs on Linguistic Reasoning [8.20398036986024]
  We introduce IOLBENCH, a novel benchmark derived from International Linguistics Olympiad (IOL) problems.<n>This dataset encompasses diverse problems testing syntax, morphology, phonology, and semantics.<n>We find that even the most advanced models struggle to handle the intricacies of linguistic complexity.
  arXiv  Detail & Related papers  (2025-01-08T03:15:10Z)
• The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model [59.357993924917]
  We study the evolution of multilingual capabilities in large language models (LLMs) during the pre-training process. We propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. We propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs.
  arXiv  Detail & Related papers  (2024-12-10T08:28:57Z)
• Large Language Models are Interpretable Learners [53.56735770834617]
  In this paper, we show a combination of Large Language Models (LLMs) and symbolic programs can bridge the gap between expressiveness and interpretability. The pretrained LLM with natural language prompts provides a massive set of interpretable modules that can transform raw input into natural language concepts. As the knowledge learned by LSP is a combination of natural language descriptions and symbolic rules, it is easily transferable to humans (interpretable) and other LLMs.
  arXiv  Detail & Related papers  (2024-06-25T02:18:15Z)
• A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers [51.8203871494146]
  The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing. Despite the breakthroughs of LLMs, the investigation into the multilingual scenario remains insufficient. This survey aims to help the research community address multilingual problems and provide a comprehensive understanding of the core concepts, key techniques, and latest developments in multilingual natural language processing based on LLMs.
  arXiv  Detail & Related papers  (2024-05-17T17:47:39Z)
• Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet? [82.02076369811402]
  Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. We present an extensive and systematic comparison of the three approaches, testing them on 6 high- and low-resource languages, three different NLU tasks, and a myriad of language and domain setups. Our observations show that supervised instruction tuning has the best trade-off between performance and resource requirements.
  arXiv  Detail & Related papers  (2024-03-04T10:48:13Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• Quantitative knowledge retrieval from large language models [4.155711233354597]
  Large language models (LLMs) have been extensively studied for their abilities to generate convincing natural language sequences. This paper explores the feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid data analysis tasks.
  arXiv  Detail & Related papers  (2024-02-12T16:32:37Z)
• FaithLM: Towards Faithful Explanations for Large Language Models [67.29893340289779]
  Large Language Models (LLMs) have become proficient in addressing complex tasks by leveraging their internal knowledge and reasoning capabilities. The black-box nature of these models complicates the task of explaining their decision-making processes. We introduce FaithLM to explain the decision of LLMs with natural language (NL) explanations.
  arXiv  Detail & Related papers  (2024-02-07T09:09:14Z)
• Machine Translation with Large Language Models: Prompt Engineering for Persian, English, and Russian Directions [0.0]
  Generative large language models (LLMs) have demonstrated exceptional proficiency in various natural language processing (NLP) tasks. We conducted an investigation into two popular prompting methods and their combination, focusing on cross-language combinations of Persian, English, and Russian.
  arXiv  Detail & Related papers  (2024-01-16T15:16:34Z)
• Let Models Speak Ciphers: Multiagent Debate through Embeddings [84.20336971784495]
  We introduce CIPHER (Communicative Inter-Model Protocol Through Embedding Representation) to address this issue. By deviating from natural language, CIPHER offers an advantage of encoding a broader spectrum of information without any modification to the model weights. This showcases the superiority and robustness of embeddings as an alternative "language" for communication among LLMs.
  arXiv  Detail & Related papers  (2023-10-10T03:06:38Z)
• Re-Reading Improves Reasoning in Large Language Models [87.46256176508376]
  We introduce a simple, yet general and effective prompting method, Re2, to enhance the reasoning capabilities of off-the-shelf Large Language Models (LLMs) Unlike most thought-eliciting prompting methods, such as Chain-of-Thought (CoT), Re2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process. We evaluate Re2 on extensive reasoning benchmarks across 14 datasets, spanning 112 experiments, to validate its effectiveness and generality.
  arXiv  Detail & Related papers  (2023-09-12T14:36:23Z)
• Chit-Chat or Deep Talk: Prompt Engineering for Process Mining [0.0]
  This research investigates the application of Large Language Models (LLMs) to augment conversational agents in process mining. We propose an innovative approach that amend many issues in existing solutions, informed by prior research on Natural Language Processing (NLP) for conversational agents. Our framework improves both accessibility and agent performance, as demonstrated by experiments on public question and data sets.
  arXiv  Detail & Related papers  (2023-07-19T11:25:12Z)
• External Reasoning: Towards Multi-Large-Language-Models Interchangeable Assistance with Human Feedback [0.0]
  This paper proposes that Large Language Models (LLMs) could be augmented through the selective integration of knowledge from external repositories. Central to this approach is the establishment of a tiered policy for textbfExternal Reasoning based on Multiple LLM Interchange Assistance. The results indicate state-of-the-art performance in crefcomparison, surpassing existing solutions including ChatPDF.com.
  arXiv  Detail & Related papers  (2023-07-05T17:05:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.