CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

• URL: http://arxiv.org/abs/2604.15663v1
• Date: Fri, 17 Apr 2026 03:35:35 GMT
• Title: CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
• Abstract summary: Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG)<n>Existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data, SVGs, schematic diagrams, and visualizations.<n>We introduce MMCoIR, the first benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, eleven libraries, and show the challenge of the task through extensive evaluation.<n>We then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code,
• Score: 16.651846645091315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML. To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, eleven libraries, and show the challenge of the task through extensive evaluation. Therefore, we then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment. CodeMMR achieves strong generalization across modalities and languages, outperforming competitive baselines (e.g., UniIR, GME, VLM2Vec) by an average of 10 points on nDCG@10. Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks, underscoring the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems. Datasets are available at HuggingFace.

Related papers

• RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices [54.956760584923295]
  Researchers have made substantial progress using Large Language Models (LLMs) for code generation.<n>However, developers typically write code based on structured designs or specifications rather than raw natural language descriptions.<n>This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate development tasks.
  arXiv  Detail & Related papers  (2026-04-24T15:35:54Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models [82.05514464090172]
  Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding.<n>However, their ability to generate code from multimodal inputs remains limited.<n>We introduce VisCodex, a unified framework that seamlessly merges vision and coding language models.
  arXiv  Detail & Related papers  (2025-08-13T17:00:44Z)
• Multilingual Multimodal Software Developer for Code Generation [35.33149292210637]
  We introduce MM-Coder, a Multilingual Multimodal software developer.<n> MM-Coder integrates visual design inputs-Unified Language (UML) diagrams and flowcharts.<n>MMEval is a new benchmark for evaluating multimodal code generation.
  arXiv  Detail & Related papers  (2025-07-11T16:19:53Z)
• CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings [32.72039589832989]
  Large language models (LLMs) have revolutionized code generation, automating programming with remarkable efficiency.<n>These advancements challenge programming skills, ethics, and assessment integrity, making the detection of LLM-generated code essential for maintaining accountability and standards.<n>We propose a framework capable of distinguishing between human- and LLM-written code across multiple programming languages, code generators, and domains.
  arXiv  Detail & Related papers  (2025-03-17T21:41:37Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters.<n>Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework.<n>Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs [9.649864680130781]
  CodeMMLU is a benchmark designed to evaluate the depth of software and code comprehension in Code Large Language Models.<n>It includes nearly 20,000 questions spanning diverse domains, including code analysis, defect detection, and software engineering principles.<n>Our evaluation reveals that even state-of-the-art models struggle with CodeMMLU.
  arXiv  Detail & Related papers  (2024-10-02T20:04:02Z)
• CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
  CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to better interpret the programming domain knowledge.<n>CodeGRAG significantly improves the code generation ability of LLMs and can even offer performance gain for cross-lingual code generation.
  arXiv  Detail & Related papers  (2024-05-03T02:48:55Z)
• IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators [49.903001442804594]
  This work investigates the prospect of leveraging compiler intermediate representations (IR) to improve the multilingual capabilities of Code-LMs. We first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files. Next, we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to learn the IR language. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across a wide variety of code generation tasks and metrics.
  arXiv  Detail & Related papers  (2024-03-06T17:52:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.