LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy

• URL: http://arxiv.org/abs/2502.11405v1
• Date: Mon, 17 Feb 2025 03:45:03 GMT
• Title: LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy
• Authors: Zhiwen Ruan, Yixia Li, He Zhu, Longyue Wang, Weihua Luo, Kaifu Zhang, Yun Chen, Guanhua Chen,
• Abstract summary: Large language models (LLMs) exhibit suboptimal performance on low-resource languages. Recent approaches have leveraged multilingual encoders alongside LLMs by introducing trainable parameters connecting the two models. We propose aname, a framework that integrates representations from all encoder layers.
• Score: 33.85811169010525
• License:
• Abstract: Despite being pretrained on multilingual corpora, large language models (LLMs) exhibit suboptimal performance on low-resource languages. Recent approaches have leveraged multilingual encoders alongside LLMs by introducing trainable parameters connecting the two models. However, these methods typically focus on the encoder's output, overlooking valuable information from other layers. We propose \aname (\mname), a framework that integrates representations from all encoder layers, coupled with the \attaname mechanism to enable layer-wise interaction between the LLM and the multilingual encoder. Extensive experiments on multilingual reasoning tasks, along with analyses of learned representations, show that our approach consistently outperforms existing baselines.

Related papers

• LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models [89.13128402847943]
  We present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision. LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks. We introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages.
  arXiv  Detail & Related papers  (2025-01-01T15:43:07Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, effectively being crosslingual? This study evaluates six state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Multilingual Large Language Models and Curse of Multilinguality [4.096453902709292]
  Large Language Models (LLMs) have gained large popularity among Natural Language Processing (NLP) researchers and practitioners. This paper navigates the landscape of multilingual LLMs, providing an introductory overview of their technical aspects. It explains underlying architectures, objective functions, pre-training data sources, and tokenization methods.
  arXiv  Detail & Related papers  (2024-06-15T11:31:39Z)
• Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners [67.85635044939836]
  Large Language Models (LLMs) have shown impressive language capabilities. In this work, we investigate the spontaneous multilingual alignment improvement of LLMs. We find that LLMs instruction-tuned on the question translation data (i.e. without annotated answers) are able to encourage the alignment between English and a wide range of languages.
  arXiv  Detail & Related papers  (2024-05-22T16:46:19Z)
• Understanding the role of FFNs in driving multilingual behaviour in LLMs [0.0]
  In this paper, we conduct an in-depth analysis of the multilingual capabilities of a family of Large Language Models. We introduce novel metrics to probe the model's multilingual behaviour at different layers and shed light on the impact of architectural choices on multilingual processing.
  arXiv  Detail & Related papers  (2024-04-22T03:47:00Z)
• Probing Multimodal Large Language Models for Global and Local Semantic Representations [57.25949445963422]
  We study which layers of Multimodal Large Language Models make the most effort to the global image information. In this study, we find that the intermediate layers of models can encode more global semantic information. We find that the topmost layers may excessively focus on local information, leading to a diminished ability to encode global information.
  arXiv  Detail & Related papers  (2024-02-27T08:27:15Z)
• SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models [97.40590590880144]
  We develop an extensive Multimodality Large Language Model (MLLM) series. We assemble a comprehensive dataset covering publicly available resources in language, vision, and vision-language tasks. We obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities.
  arXiv  Detail & Related papers  (2024-02-08T18:59:48Z)
• Probing LLMs for Joint Encoding of Linguistic Categories [10.988109020181563]
  We propose a framework for testing the joint encoding of linguistic categories in Large Language Models (LLMs) We find evidence of joint encoding both at the same (related part-of-speech (POS) classes) and different (POS classes and related syntactic dependency relations) levels of linguistic hierarchy.
  arXiv  Detail & Related papers  (2023-10-28T12:46:40Z)
• Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training [21.934439663979663]
  Multi-branch multilingual language model (MBLM) built on Multilingual pre-trained language models (MPLMs) Method based on transferring knowledge from high-performance monolingual models with a teacher-student framework. Results on two cross-lingual classification tasks show that, with only the task's supervised data used, our method improves both the supervised and zero-shot performance of MPLMs.
  arXiv  Detail & Related papers  (2022-02-28T09:51:32Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.