FLeX: Fourier-based Low-rank EXpansion for multilingual transfer

• URL: http://arxiv.org/abs/2604.06253v1
• Date: Mon, 06 Apr 2026 19:26:13 GMT
• Title: FLeX: Fourier-based Low-rank EXpansion for multilingual transfer
• Abstract summary: Cross-lingual code generation is critical in enterprise environments where multiple programming languages coexist.<n>This paper investigates whether fine-tuning methods and parameter enhancements can improve cross-lingual transfer from Python to languages like Java.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-lingual code generation is critical in enterprise environments where multiple programming languages coexist. However, fine-tuning large language models (LLMs) individually for each language is computationally prohibitive. This paper investigates whether parameter-efficient fine-tuning methods and optimizer enhancements can improve cross-lingual transfer from Python to languages like Java. We fine-tune the Code Llama 7B model using low-rank adaptation (LoRA) to optimize a small subset of parameters and compare Adam and Sophia optimizers, while exploring a novel Fourier-based regularization technique. Our contributions include: (1)demonstrating that LoRA fine-tuning on a small, high-quality dataset (MBPP) can exceed the pass@1 performance of the more broadly fine-tuned Code Llama-Python-7B model (40.1% vs. 38.4%); (2) showing that while Sophia achieves faster convergence than Adam, final pass@1 scores show marginal differences; and (3) presenting evidence that Fourier-based regularization during fine-tuning significantly improves cross-lingual transfer, achieving 42.1% pass@1 on Java tasks compared to the 34.2% baseline. These findings suggest that combining LoRA, optimized training methods, and frequency-domain regularization can efficiently adapt single-language LLMs to perform well across multiple programming languages.

Related papers

• Online Language Adaptive Sampling for Better Distributed Cross-lingual Gains [5.745735995922634]
  Realignment is a promising approach for improving the cross-lingual transfer ability of multilingual language models.<n>We propose a simple yet effective adaptive sampling strategy that assigns trainable sampling probabilities to each language.<n>Our method employs an inner-outer optimization loop with a small overhead, leading to consistent performance improvements.
  arXiv  Detail & Related papers  (2026-09-14T03:23:26Z)
• Scaling Laws for Code: Every Programming Language Matters [73.6302896079007]
  Code large language models (Code LLMs) are powerful but costly to train.<n>Different programming languages (PLs) have varying impacts during pre-training that significantly affect base model performance.<n>We present the first systematic exploration of scaling laws for multilingual code pre-training.
  arXiv  Detail & Related papers  (2025-12-15T16:07:34Z)
• Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation [23.597327427570658]
  We present a comprehensive study of three prefix-based methods for zero-shot cross-lingual transfer from English to 35+ high- and low-resource languages.<n>With Llama 3.1 8B, prefix methods outperform LoRA-baselines by up to 6% on the Belebele benchmark.<n>Similar improvements were observed with Mistral v0.3 7B as well.
  arXiv  Detail & Related papers  (2025-10-28T16:48:03Z)
• Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages [4.702593857707973]
  Large Language Models (LLMs) excel in English, but their performance degrades significantly on low-resource languages (LRLs) due to English-centric training.<n>We propose a novel architecture that fuses all intermediate layers, enriching the linguistic information passed to the LLM.<n>We observe strong performance gains in LRLs, improving Sinhala classification accuracy from 71.66% to 75.86% and achieving clear improvements across Indic languages such as Tamil, Bengali, and Malayalam.
  arXiv  Detail & Related papers  (2025-08-12T17:17:13Z)
• NL in the Middle: Code Translation with LLMs and Intermediate Representations [56.77064674776534]
  Large language models (LLMs) produce buggy code translations.<n>One promising avenue to improve translation accuracy is through intermediate representations.<n>We investigate whether LLM-based code translation can benefit from intermediate representations.
  arXiv  Detail & Related papers  (2025-07-11T14:29:21Z)
• Unraveling the Potential of Large Language Models in Code Translation: How Far Are We? [4.616570111453259]
  Large language models (LLMs) exhibit state-of-the-art performance in various tasks, but struggle for code translation. We conduct a large-scale empirical study to exploit the capabilities and incapabilities of LLMs in code translation tasks. We propose two methods: (1) intermediary translation which selects an intermediary language between the source and target ones; and (2) self-training which fine-tunes LLMs on self-generated parallel data.
  arXiv  Detail & Related papers  (2024-10-13T12:20:12Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead.<n>To optimize the pruning process itself, only thresholds are communicated between a server and clients instead of parameters.<n>Global thresholds are used to update model parameters by extracting aggregated parameter importance.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes [53.4856038354195]
  Pre-trained large language models (LLMs) need fine-tuning to improve their responsiveness to natural language instructions. FedKSeed employs zeroth-order optimization with a finite set of random seeds. It significantly reduces transmission requirements between the server and clients to just a few random seeds.
  arXiv  Detail & Related papers  (2023-12-11T13:03:21Z)
• OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval [91.76575626229824]
  We present OneAligner, an alignment model specially designed for sentence retrieval tasks. When trained with all language pairs of a large-scale parallel multilingual corpus (OPUS-100), this model achieves the state-of-the-art result. We conclude through empirical results and analyses that the performance of the sentence alignment task depends mostly on the monolingual and parallel data size.
  arXiv  Detail & Related papers  (2022-05-17T19:52:42Z)
• Distributionally Robust Multilingual Machine Translation [94.51866646879337]
  We propose a new learning objective for Multilingual neural machine translation (MNMT) based on distributionally robust optimization. We show how to practically optimize this objective for large translation corpora using an iterated best response scheme. Our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.
  arXiv  Detail & Related papers  (2021-09-09T03:48:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.