When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models

• URL: http://arxiv.org/abs/2606.01671v1
• Date: Mon, 01 Jun 2026 04:28:44 GMT
• Title: When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models
• Abstract summary: This paper showcases the navigation of retaining figurative and cultural semantics in low-resource Southeast Asian languages such as Hindi, Bengali, and Thai.<n>To tackle such complexity, we present Varnika, a reconstructed multimodal corpus comprising 3,533 multilingual idioms.
• Score: 11.355899871129559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the contemporary epoch of multilingual education, learning idioms provides a fascinating gateway towards creativity, cultural values, historical context, and diverse perspectives inherent to various linguistic traditions. This paper showcases the navigation of retaining figurative and cultural semantics in low-resource Southeast Asian languages such as Hindi, Bengali, and Thai, where culturally rich idioms pose significant obstacles for computational modeling and cross-linguistic transfer due to their deep metaphorical complexity. To tackle such complexity, we present Varnika, a reconstructed multimodal idiom corpus comprising 3,533 multilingual idioms, enriched with seven idiomatic tones aligned with both textual and visual representations. Additionally, to infer informative idiomatic understanding, we introduce a Hybrid Mixture-of-Experts (HybridMoE) framework that embeds multiple idiomatic expert opinions while mitigating expert sparsity by integrating outputs from both selected and unselected experts through controlled hybridization, further augmented with Idiomatic Property Signals via masked multimodal embeddings. To analyze the performance across multiple dimensions, we propose the IDIO-TONE and Idiomatic Validation Score, a three-stage evaluation pipeline measuring (i) literal translation fidelity, (ii) visual-semantic alignment, and (iii) idiomatic meaning retention. Empirical evaluations highlight that HybridMoE achieves 5--6\% performance gains across advanced vision language models, demonstrating improved representation of figurative language and culturally embedded meaning in multilingual multimodal settings

Related papers

• Wisdom in Unity: The Role of Multilingual Training in Figurative Language Identification in Proverbs [0.0]
  We evaluate five models, including multilingual encoders and instruction tuned LLMs, under progressively increasing levels of multilingual supervision.<n>We introduce a multidimensional annotation framework for proverbs that characterizes them through four complementary figurative forms.<n>Our findings show that approximately 50% of the translated multilingual training data is sufficient to achieve near-optimal figurative language identification performance.
  arXiv  Detail & Related papers  (2026-08-08T12:15:52Z)
• A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding [15.171586338601522]
  Potentially idiomatic expressions (PIEs) construe meanings inherently tied to the everyday experience of a language community.<n>We present XMPIE, a parallel multilingual and multimodal dataset of potentially idiomatic expressions.
  arXiv  Detail & Related papers  (2026-01-13T15:20:28Z)
• M3DR: Towards Universal Multilingual Multimodal Document Retrieval [0.0]
  We present M3DR (Multilingual Multimodal Document Retrieval), a framework designed to bridge this gap across languages.<n>We generalize across different vision-language architectures and model sizes, enabling robust cross-lingual and cross-modal alignment.<n>Our models, NetraEmbed and ColNetraEmbed achieve state-of-the-art performance with 150% relative improvements on cross-lingual retrieval.
  arXiv  Detail & Related papers  (2025-12-03T07:17:59Z)
• MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation [91.22008265721952]
  MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned benchmark covering 8 Asian countries and 10 languages.<n>This is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech.<n>We propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity.
  arXiv  Detail & Related papers  (2025-10-07T14:12:12Z)
• Decoding Memes: Benchmarking Narrative Role Classification across Multilingual and Multimodal Models [26.91963265869296]
  This work investigates the challenging task of identifying narrative roles in Internet memes.<n>It builds on an annotated dataset originally skewed toward the 'Other' class.<n> Comprehensive lexical and structural analyses highlight the nuanced, culture-specific, and context-rich language used in real memes.
  arXiv  Detail & Related papers  (2025-06-29T07:12:11Z)
• JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human Self-Destructive Behavior Content in Jirai Community [9.492476871323763]
  This paper introduces JiraiBench, the first bilingual benchmark for evaluating large language models' effectiveness in detecting self-destructive content.<n>We focus on the transnational "Jirai" (landmine) online subculture that encompasses multiple forms of self-destructive behaviors including drug overdose, eating disorders, and self-harm.<n>Our dataset comprises 10,419 Chinese posts and 5,000 Japanese posts with multidimensional annotation along three behavioral categories.
  arXiv  Detail & Related papers  (2025-03-27T16:48:58Z)
• Multi-lingual and Multi-cultural Figurative Language Understanding [69.47641938200817]
  Figurative language permeates human communication, but is relatively understudied in NLP. We create a dataset for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region. All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data.
  arXiv  Detail & Related papers  (2023-05-25T15:30:31Z)
• LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation [94.33019040320507]
  Multimodal Machine Translation (MMT) focuses on enhancing text-only translation with visual features. Recent advances still struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases. We propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages.
  arXiv  Detail & Related papers  (2022-10-19T12:21:39Z)
• Revamping Multilingual Agreement Bidirectionally via Switched Back-translation for Multilingual Neural Machine Translation [107.83158521848372]
  multilingual agreement (MA) has shown its importance for multilingual neural machine translation (MNMT) We present textbfBidirectional textbfMultilingual textbfAgreement via textbfSwitched textbfBack-textbftranslation (textbfBMA-SBT) It is a novel and universal multilingual agreement framework for fine-tuning pre-trained MNMT models.
  arXiv  Detail & Related papers  (2022-09-28T09:14:58Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training [119.16007395162431]
  M3P is a Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training. We show that M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.
  arXiv  Detail & Related papers  (2020-06-04T03:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.