Language-Conditioned Visual Grounding with CLIP Multilingual

• URL: http://arxiv.org/abs/2605.09060v1
• Date: Sat, 09 May 2026 17:06:16 GMT
• Title: Language-Conditioned Visual Grounding with CLIP Multilingual
• Abstract summary: Cross-language divergence could arise from the visual encoder, the text branch, or their interaction.<n>We resolve this ambiguity through a dense multilingual CLIP probe in which the visual encoder is held identical across thirteen typologically diverse languages.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multilingual vision-language models exhibit systematic performance gaps across languages, but the mechanism remains ambiguous: cross-language divergence could arise from the visual encoder, the text branch, or their interaction. We resolve this ambiguity through a dense multilingual CLIP probe in which the visual encoder is held identical across thirteen typologically diverse languages and only the XLM-RoBERTa text branch varies. We evaluate two CLIP architectures spanning a 7x visual-encoder scale gap (XLM-R base + ViT-B/32, ~87M visual parameters; XLM-R large + ViT-H/14, ~632M) on 11 concepts and 210 images, and quantify cross-language agreement via cluster-mask IoU, top-percentile IoU, and Spearman rank correlation against an English reference (n=2,310 paired observations per language). Three findings emerge. First, low-resource languages (Arabic, Basque, Luxembourgish) incur a structural penalty at both backbone scales (Wilcoxon HR>LR p<10^-300; cluster-mask IoU gap +0.114 at base, +0.143 at large), isolating the deficit to the text branch. Second, scaling the encoder 7x widens the gap for structural failure cases (Basque Δ=-0.056, Luxembourgish Δ=-0.076) while improving Arabic (Δ=+0.033), separating corpus-coverage from tokeniser-fertility failures. Third, peak similarity is preserved across languages (mean ratio 0.94 at large scale) while cluster-mask IoU drops sharply, identifying spatial misalignment, not signal collapse, as the dominant failure mode. At 3.4-3.9 Wh per 1,000 queries, dense-CLIP grounding is competitive with high-throughput inference budgets, positioning it as a practical substrate for energy-aware multilingual deployment.

Related papers

• Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning [80.05116806217397]
  Large Language Model (LLM) unlearning aims to suppress target knowledge while preserving general capabilities.<n>We introduce CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings.<n>Common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a designated language.
  arXiv  Detail & Related papers  (2026-09-05T08:26:15Z)
• Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix [0.0]
  We characterize a sufficient set of three constraints that prevent the failure of a Gumbel-softmax-based discrete symbol bottleneck.<n>We demonstrate zero symbol collapse in all 32 seeds, with the blackboard achieving 79-100% semantic binding across three encoder architectures.
  arXiv  Detail & Related papers  (2026-07-09T09:55:17Z)
• Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation [0.0]
  Cross-attention patterns in neural machine translation are widely used to study how multilingual models align linguistic structure.<n>We report a systematic artifact in cross-attention analysis of NLLB-200 (600M)<n>We term these "attention sinks," extending findings from LLMs to NMT cross-attention and identifying a causal mechanism rooted in vocabulary design rather than position bias.
  arXiv  Detail & Related papers  (2026-05-02T03:59:00Z)
• MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes [0.0]
  This paper presents our system for the CHiPSAL 2026 shared task, addressing both Subtask A and Subtask B.<n>We propose a hybrid cross-modal attention fusion architecture that combines CLIP for visual encoding with BGE-M3 for multilingual text representation.<n>We show explicit cross-modal reasoning achieves a 5.9% F1-macro improvement over text-only baselines on Subtask A.
  arXiv  Detail & Related papers  (2026-04-13T07:37:14Z)
• Bilingual Text-to-Motion Generation: A New Benchmark and Baselines [52.71312720094036]
  We introduce BiHumanML3D, the first bilingual text-to-motion benchmark, constructed via LLM-assisted annotation and rigorous manual correction.<n>We also propose a simple yet effective baseline, Bilingual Motion Diffusion synthesis (BiMD), featuring Cross-Lingual Alignment (CLA)<n>BiMD with CLA achieves an FID of 0.045 vs. 0.169 and R@3 of 82.8% vs. 80.8%, significantly outperforms monolingual diffusion models and translation baselines on BiHumanML3D.
  arXiv  Detail & Related papers  (2026-03-26T08:48:27Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• Comprehension of Multilingual Expressions Referring to Target Objects in Visual Inputs [47.944645462877894]
  Referring Expression (REC) requires models to localize objects in images based on natural language descriptions.<n>This work addresses multilingual REC through two main contributions.<n>We construct a unified multilingual dataset spanning 10 languages, by systematically expanding 12 existing English REC benchmarks through machine translation and context-based translation enhancement.<n>The resulting dataset comprises approximately 8 million multilingual referring expressions across 177,620 images, with 336,882 annotated objects.
  arXiv  Detail & Related papers  (2025-11-14T15:54:34Z)
• Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector [25.65114670027799]
  Learned Sparse Retrieval (LSR) combines the efficiency of bi-encoders with the transparency of lexical matching.<n>We introduce MILCO, an LSR architecture that maps queries and documents from different languages into a shared English lexical space.
  arXiv  Detail & Related papers  (2025-10-01T08:58:25Z)
• On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation [104.85258654917297]
  We find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance. We propose Language Aware Vocabulary Sharing (LAVS) to construct the multilingual vocabulary. We conduct experiments on a multilingual machine translation benchmark in 11 languages.
  arXiv  Detail & Related papers  (2023-05-18T12:43:31Z)
• Inducing Language-Agnostic Multilingual Representations [61.97381112847459]
  Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. We examine three approaches for this: (i) re-aligning the vector spaces of target languages to a pivot source language; (ii) removing language-specific means and variances, which yields better discriminativeness of embeddings as a by-product; and (iii) increasing input similarity across languages by removing morphological contractions and sentence reordering.
  arXiv  Detail & Related papers  (2020-08-20T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.