MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost

• URL: http://arxiv.org/abs/2412.01271v1
• Date: Mon, 02 Dec 2024 08:38:19 GMT
• Title: MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
• Abstract summary: We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency.<n>We introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model.
• Score: 39.43930856789352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages. Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (38.61 for English vs. 37.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases.

Related papers

• uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data [3.364569898365253]
  We propose a lightweight and data-efficient framework for multilingual vision-language alignment.<n>Our approach requires no image-text pairs or text-text pairs and freezes both the pretrained image encoder and multilingual text encoder during training.<n>This minimal training setup enables robust multilingual alignment even for languages with limited supervision.
  arXiv  Detail & Related papers  (2025-11-17T06:34:49Z)
• CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning [7.439550425786999]
  We introduce CONCAP, a multilingual image captioning model that integrates retrieved captions with image-specific concepts.<n>Experiments on the XM3600 dataset indicate that CONCAP enables strong performance on low- and mid-resource languages.
  arXiv  Detail & Related papers  (2025-07-27T21:00:02Z)
• NeoBabel: A Multilingual Open Tower for Visual Generation [32.79724699684266]
  We introduce NeoBabel, a novel multilingual image generation framework.<n>It supports six languages: English, Chinese, Dutch, French, Hindi, and Persian.<n>It achieves state-of-the-art multilingual performance while retaining strong English capability.
  arXiv  Detail & Related papers  (2025-07-08T16:19:45Z)
• Florenz: Scaling Laws for Systematic Generalization in Vision-Language Models [17.444066202370397]
  Cross-lingual transfer enables vision-language models to perform vision tasks in various languages with training data only in one language. Current approaches rely on large pre-trained multilingual language models. We propose Florenz, a monolingual encoder-decoder VLM with 0.4B to 11.2B parameters combining the pre-trained VLM Florence-2 and the large language model Gemma-2.
  arXiv  Detail & Related papers  (2025-03-12T14:41:10Z)
• mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus [52.83121058429025]
  We introduce mOSCAR, the first large-scale multilingual and multimodal document corpus crawled from the web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. It shows a strong boost in few-shot learning performance across various multilingual image-text tasks and benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T00:13:32Z)
• A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene [11.265838907079196]
  We propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context.
  arXiv  Detail & Related papers  (2024-04-17T10:56:06Z)
• PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation [6.528169059491259]
  We propose a plug-and-play language transfer method based on knowledge distillation. All we need to do is train a lightweight-like parameter-efficient adapter (PEA) with only 6M parameters under teacher knowledge distillation. We are surprised to find that freezing the parameters of UNet can still achieve remarkable performance on the language-specific prompt evaluation set.
  arXiv  Detail & Related papers  (2023-11-28T02:31:52Z)
• Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages [76.35234803589412]
  MPM is an effective training paradigm for training large multimodal models in non-English languages. We build large multimodal models VisCPM in image-to-text and text-to-image generation, which achieve state-of-the-art (open-source) performance in Chinese.
  arXiv  Detail & Related papers  (2023-08-23T09:55:41Z)
• mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs [50.17767479660832]
  Vision-language models (Vision-LLMs) align pretrained image encoders with (frozen) large language models (LLMs) and post-hoc condition LLMs to understand' the image input. We present mBLIP, the first Vision-LLM leveraging multilingual LLMs, which we obtain in a computationally efficient manner on consumer-level hardware.
  arXiv  Detail & Related papers  (2023-07-13T17:51:58Z)
• Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations [53.89380284760555]
  We introduce Babel-ImageNet, a massively multilingual benchmark that offers partial translations of ImageNet labels to 100 languages. We evaluate 11 public multilingual CLIP models on our benchmark, demonstrating a significant gap between English ImageNet performance and that of high-resource languages. We show that the performance of multilingual CLIP can be drastically improved for low-resource languages with parameter-efficient language-specific training.
  arXiv  Detail & Related papers  (2023-06-14T17:53:06Z)
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training [52.852163987208826]
  UC2 is the first machine translation-augmented framework for cross-lingual cross-modal representation learning. We propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM) Our proposed framework achieves new state-of-the-art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.
  arXiv  Detail & Related papers  (2021-04-01T08:30:53Z)
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models [144.85290716246533]
  We study zero-shot cross-lingual transfer of vision-language models. We propose a Transformer-based model that learns contextualized multilingual multimodal embeddings.
  arXiv  Detail & Related papers  (2021-03-16T04:37:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.