Analysis of AdvFusion: Adapter-based Multilingual Learning for Code Large Language Models

• URL: http://arxiv.org/abs/2511.02869v1
• Date: Mon, 03 Nov 2025 23:45:27 GMT
• Title: Analysis of AdvFusion: Adapter-based Multilingual Learning for Code Large Language Models
• Authors: Amirreza Esmaeili, Fahd Seddik, Yongyi Ji, Fatemeh Fard, Fuxiang Chen,
• Abstract summary: AdapterFusion aims to enhance task performance by leveraging information from multiple programming languages.<n>In this study, we investigated AdvFusion on Code Large Language Models (Code-LLMs)<n>In code generation, AdvFusion outperformed AdapterFusion but not other PEFT methods (LoRA, Compacter, and TaskAdapter)<n>In commit message generation, AdapterFusion performed better than AdvFusion, and contrary to code generation, we found that the other PEFT methods do not have better performance.
• Score: 0.043496401697112684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Programming languages can benefit from one another by utilizing a language model for software engineering tasks. Full fine-tuning and Parameter Efficient Fine-Tuning (PEFT) of Code Language Models (Code-LMs) has been explored for multilingual knowledge transfer. AdapterFusion is a PEFT architecture that aims to enhance task performance by leveraging information from multiple programming languages, but primarily focuses on the target programming language. In our previous work, we proposed AdvFusion, a novel PEFT-based approach that effectively learns from other programming languages before adapting to the target task. Though previous experiments showed that AdvFusion outperformed AdapterFusion and LoRA, it was applied on pre-trained Code-LMs and was limited to only two tasks, code summarization and method name prediction. In this study, we expanded our work and investigated AdvFusion on Code Large Language Models (Code-LLMs), considering three new tasks: code generation, code translation, and commit message generation. We observed that different Code-LLMs/tasks exhibit different characteristics. In code generation, AdvFusion outperformed AdapterFusion but not other PEFT methods (LoRA, Compacter, and TaskAdapter). In commit message generation, AdapterFusion performed better than AdvFusion, and contrary to code generation, we found that the other PEFT methods do not have better performance. In code translation, AdvFusion performed worse than AdapterFusion overall, with the performance gap marginally widening as the model size increases. However, consistent with code generation, other PEFT methods showed better performance.

Related papers

• Scaling Laws for Code: Every Programming Language Matters [73.6302896079007]
  Code large language models (Code LLMs) are powerful but costly to train.<n>Different programming languages (PLs) have varying impacts during pre-training that significantly affect base model performance.<n>We present the first systematic exploration of scaling laws for multilingual code pre-training.
  arXiv  Detail & Related papers  (2025-12-15T16:07:34Z)
• I Can't Share Code, but I need Translation -- An Empirical Study on Code Translation through Federated LLM [3.9373541926236766]
  This study demonstrates that participants can collaboratively develop a FedLLM for efficient code translation.<n>Our findings indicate that FedLLM offers a collaborative approach to code translation and could serve as a promising direction for future research in this field.
  arXiv  Detail & Related papers  (2025-01-10T05:43:36Z)
• Unraveling the Potential of Large Language Models in Code Translation: How Far Are We? [4.616570111453259]
  Large language models (LLMs) exhibit state-of-the-art performance in various tasks, but struggle for code translation. We conduct a large-scale empirical study to exploit the capabilities and incapabilities of LLMs in code translation tasks. We propose two methods: (1) intermediary translation which selects an intermediary language between the source and target ones; and (2) self-training which fine-tunes LLMs on self-generated parallel data.
  arXiv  Detail & Related papers  (2024-10-13T12:20:12Z)
• Cool-Fusion: Fuse Large Language Models without Training [73.17551121242602]
  Cool-Fusion fuses the knowledge of source LLMs, which does not require training.<n>Experiments have been conducted across a variety of benchmark datasets.<n>On GSM8K, Cool-Fusion increases accuracy from three strong source LLMs by a significant margin of 17.4%.
  arXiv  Detail & Related papers  (2024-07-29T09:02:19Z)
• IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators [49.903001442804594]
  This work investigates the prospect of leveraging compiler intermediate representations (IR) to improve the multilingual capabilities of Code-LMs. We first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files. Next, we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to learn the IR language. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across a wide variety of code generation tasks and metrics.
  arXiv  Detail & Related papers  (2024-03-06T17:52:08Z)
• CoLLiE: Collaborative Training of Large Language Models in an Efficient Way [59.09824823710863]
  CoLLiE is an efficient library that facilitates collaborative training of large language models. With its modular design and comprehensive functionality, CoLLiE offers a balanced blend of efficiency, ease of use, and customization.
  arXiv  Detail & Related papers  (2023-12-01T08:02:16Z)
• CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model [58.127534002232096]
  This paper introduces CodeFuse-13B, an open-sourced pre-trained code LLM. It is specifically designed for code-related tasks with both English and Chinese prompts. CodeFuse achieves its effectiveness by utilizing a high quality pre-training dataset.
  arXiv  Detail & Related papers  (2023-10-10T02:38:44Z)
• AdvFusion: Adapter-based Knowledge Transfer for Code Summarization on Code Language Models [0.3228451873135423]
  We propose AdvFusion, a PEFT-based approach that effectively learns from other languages before adapting to the target task.<n>We evaluate it on code summarization and method name prediction.<n>It outperforms AdapterFusion by up to 1.7 points and surpasses LoRA with gains of 1.99, 1.26, and 2.16 for Ruby, JavaScript, and Go, respectively.
  arXiv  Detail & Related papers  (2023-07-15T17:17:16Z)
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models [75.25782573728677]
  This paper presents a framework for adapter-based parameter-efficient fine-tuning (PEFT) of language models (LLMs) The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning.
  arXiv  Detail & Related papers  (2023-04-04T16:31:37Z)
• On The Cross-Modal Transfer from Natural Language to Code through Adapter Modules [0.0]
  We explore the knowledge transfer using adapters in software engineering. Three programming languages, C/C++, Python, and Java, are studied along with extensive experiments on the best setup used for adapters. Our results can open new directions to build smaller models for more software engineering tasks.
  arXiv  Detail & Related papers  (2022-04-19T04:18:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.