MLIR-Smith: A Novel Random Program Generator for Evaluating Compiler Pipelines

• URL: http://arxiv.org/abs/2601.02218v1
• Date: Mon, 05 Jan 2026 15:43:09 GMT
• Title: MLIR-Smith: A Novel Random Program Generator for Evaluating Compiler Pipelines
• Authors: Berke Ates, Filip Dobrosavljević, Theodoros Theodoridis, Zhendong Su,
• Abstract summary: We introduce MLIR-Smith, a novel random program generator specifically designed to test and evaluate compiler optimizations.<n>By providing a tool that can generate random MLIR programs, this paper enhances our ability to evaluate and improve compilers.
• Score: 5.268554895844063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compilers are essential for the performance and correct execution of software and hold universal relevance across various scientific disciplines. Despite this, there is a notable lack of tools for testing and evaluating them, especially within the adaptable Multi-Level Intermediate Representation (MLIR) context. This paper addresses the need for a tool that can accommodate MLIR's extensibility, a feature not provided by previous methods such as Csmith. Here we introduce MLIR-Smith, a novel random program generator specifically designed to test and evaluate MLIR-based compiler optimizations. We demonstrate the utility of MLIR-Smith by conducting differential testing on MLIR, LLVM, DaCe, and DCIR, which led to the discovery of multiple bugs in these compiler pipelines. The introduction of MLIR-Smith not only fills a void in the realm of compiler testing but also emphasizes the importance of comprehensive testing within these systems. By providing a tool that can generate random MLIR programs, this paper enhances our ability to evaluate and improve compilers and paves the way for future tools, potentially shaping the wider landscape of software testing and quality assurance.

Related papers

• Environment-Aware Code Generation: How far are We? [52.69113158357018]
  It is unclear whether large language models (LLMs) can reliably generate executable code tailored to a user's specific environment.<n>We present the first systematic study of Environment-Aware Code Generation (EACG), where generated code must be functionally correct and directly executable under arbitrary software configurations.<n>Our results show that current LLMs struggle with environment-specific code generation, while our adaptations improve environment compatibility and executability.
  arXiv  Detail & Related papers  (2026-01-18T04:58:15Z)
• An Agentic Framework for Autonomous Materials Computation [70.24472585135929]
  Large Language Models (LLMs) have emerged as powerful tools for accelerating scientific discovery.<n>Recent advances integrate LLMs into agentic frameworks, enabling retrieval, reasoning, and tool use for complex scientific experiments.<n>Here, we present a domain-specialized agent designed for reliable automation of first-principles materials computations.
  arXiv  Detail & Related papers  (2025-12-22T15:03:57Z)
• ML-Tool-Bench: Tool-Augmented Planning for ML Tasks [23.54937738755734]
  We introduce a benchmark for evaluating tool-augmented machine learning agents.<n>Our benchmark goes beyond traditional tool-use evaluation by incorporating an in-memory named object management.<n>Our approach improves over ReAct by 16.52 percentile positions, taking the median across all Kaggle challenges.
  arXiv  Detail & Related papers  (2025-11-29T23:59:40Z)
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments [70.42705564227548]
  We propose an automated environment construction pipeline for large language models (LLMs)<n>This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools.<n>We also introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution.
  arXiv  Detail & Related papers  (2025-08-12T09:45:19Z)
• Machine Learning Pipeline for Software Engineering: A Systematic Literature Review [0.0]
  This systematic literature review examines state-of-the-art Machine Learning pipelines designed for software engineering (SE)<n>Our findings show that robust preprocessing, such as SMOTE for data balancing, improves model reliability.<n> Ensemble methods like Random Forest and Gradient Boosting dominate performance across tasks.<n>New metrics like Best Arithmetic Mean (BAM) are emerging in niche applications.
  arXiv  Detail & Related papers  (2025-07-31T15:37:30Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• LLM Agents Making Agent Tools [2.5529148902034637]
  Tool use has turned large language models (LLMs) into powerful agents that can perform complex multi-step tasks.<n>But these tools must be implemented in advance by human developers.<n>We propose ToolMaker, an agentic framework that autonomously transforms papers with code into LLM-compatible tools.
  arXiv  Detail & Related papers  (2025-02-17T11:44:11Z)
• SnipGen: A Mining Repository Framework for Evaluating LLMs for Code [51.07471575337676]
  Language Models (LLMs) are trained on extensive datasets that include code repositories.<n> evaluating their effectiveness poses significant challenges due to the potential overlap between the datasets used for training and those employed for evaluation.<n>We introduce SnipGen, a comprehensive repository mining framework designed to leverage prompt engineering across various downstream tasks for code generation.
  arXiv  Detail & Related papers  (2025-02-10T21:28:15Z)
• Learning to Ask: When LLM Agents Meet Unclear Instruction [55.65312637965779]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone.<n>We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench.<n>We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• Large Language Models to Generate System-Level Test Programs Targeting Non-functional Properties [3.3305233186101226]
  This paper proposes Large Language Models (LLMs) to generate test programs. We take a first glance at how pre-trained LLMs perform in test program generation to optimize non-functional properties of the DUT.
  arXiv  Detail & Related papers  (2024-03-15T08:01:02Z)
• Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study [72.24266814625685]
  We explore the performance of large language models (LLMs) across the entire software development lifecycle with DevEval.<n>DevEval features four programming languages, multiple domains, high-quality data collection, and carefully designed and verified metrics for each task.<n> Empirical studies show that current LLMs, including GPT-4, fail to solve the challenges presented within DevEval.
  arXiv  Detail & Related papers  (2024-03-13T15:13:44Z)
• ML-driven Hardware Cost Model for MLIR [1.2987894327817158]
  We develop a machine learning-based cost model for high-level MLIR. By considering the incoming MLIR as a text input a la NLP models we can apply well-known techniques from modern NLP research. We show that these models can provide reasonably good estimates with low error bounds for various hardware characteristics of interest.
  arXiv  Detail & Related papers  (2023-02-14T11:32:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.