Fuzzing MLIR Compilers with Custom Mutation Synthesis

• URL: http://arxiv.org/abs/2404.16947v2
• Date: Tue, 27 Aug 2024 16:08:52 GMT
• Title: Fuzzing MLIR Compilers with Custom Mutation Synthesis
• Abstract summary: We develop a new test generator called SYNTHFUZZ that combines grammar-based fuzzing with custom synthesis mutation. It obviates the need to manually define custom mutation operators for each dialect. Our evaluation shows that SYNTHFUZZ on average improves MLIR dialect pair coverage by 1.75 times, which increases branch coverage by 1.22 times.
• Score: 6.617861009996863
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compiler technologies in deep learning and domain-specific hardware acceleration are increasingly adopting extensible compiler frameworks such as Multi-Level Intermediate Representation (MLIR) to facilitate more efficient development. With MLIR, compiler developers can easily define their own custom IRs in the form of MLIR dialects. However, the diversity and rapid evolution of such custom IRs make it impractical to manually write a custom test generator for each dialect. To address this problem, we design a new test generator called SYNTHFUZZ that combines grammar-based fuzzing with custom mutation synthesis. The key essence of SYNTHFUZZ is two fold: (1) It automatically infers parameterized context-dependent custom mutations from existing test cases. (2) It then concretizes the mutation's content depending on the target context and reduces the chance of inserting invalid edits by performing k-ancestor and pre(post)fix matching. SYNTHFUZZ obviates the need to manually define custom mutation operators for each dialect. We compare SYNTHFUZZ to three baselines: Grammarinator, MLIRSmith, and NeuRI. We conduct this comprehensive comparison on four different MLIR projects. Each project defines a new set of MLIR dialects where manually writing a custom test generator would take weeks of effort. Our evaluation shows that SYNTHFUZZ on average improves MLIR dialect pair coverage by 1.75 times, which increases branch coverage by 1.22 times. Further, we show that our context dependent custom mutation increases the proportion of valid tests by up to 1.11 times, indicating that SYNTHFUZZ correctly concretizes its parameterized mutations with respect to the target context. Parameterization of the mutations reduces the fraction of tests violating the base MLIR constraints by 0.57 times, increasing the time spent fuzzing dialect-specific code.

Related papers

• Dictionary-Guided Mutation Operators for Automated HDL Repair [0.8450904497835263]
  We propose a dictionary-guided HDL repair system that combines ANTLR-derived DUT-specific mutation vocabularies with a simulation-divergence fault localization module.<n>The proposed approach produces correct oracle-passing repairs on 14 bug variants, including a 6-edit multi-bug instance that CirFix cannot repair.
  arXiv  Detail & Related papers  (2026-09-01T18:43:53Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation [53.844308305341166]
  We propose textbfAgentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing.<n> Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors.
  arXiv  Detail & Related papers  (2026-05-28T06:23:31Z)
• A semantic mutation metric for metamorphic relation adequacy in scientific computing programs [6.691613382732758]
  We propose the Semantic Mutation Score (SMS), built on five domain-semantic operators.<n>SMS degenerates almost everywhere to MS in a characterised limit.<n> SMS is a backward-compatible adequacy metric for domain-semantic-relation sets in scientific computing.
  arXiv  Detail & Related papers  (2026-05-17T13:16:13Z)
• From I/O to Code with Discovery Agent [103.88427301265669]
  We propose DIO-Agent, a discovery agent for IO2Code.<n>Our method frames IO2Code as an evolutionary search over discrete program space.<n>Extensive experiments show that DIO-Agent consistently outperforms both traditional program-by-example method and SOTA evolution-agent baselines.
  arXiv  Detail & Related papers  (2026-05-14T18:57:32Z)
• TENSURE: Fuzzing Sparse Tensor Compilers (Registered Report) [9.083996936429465]
  We present TENSURE, the first black-box fuzzing framework specifically designed for the testing of sparse Compilers.<n>We propose a novel constraint-based generation algorithm that guarantees 100% semantic validity of synthesized kernels.<n>Our evaluation on two state-of-the-art systems, TACO and Finch, reveals widespread fragility, particularly in TACO, where TENSURE exposed crashes or silent miscompilations.
  arXiv  Detail & Related papers  (2026-03-19T00:13:14Z)
• A Declarative Framework for Hand-Crafted Mutation Analysis and Management [0.0]
  We present a declarative framework for hand-crafted mutation analysis and management.<n>We describe five mutation representations: comment-based, preprocessor-based, patch-based, match-and-replace, and in-AST mutations.<n>We implement these ideas in Marauder, a prototype system for injecting, activating, resetting, and composing hand-crafted mutants.
  arXiv  Detail & Related papers  (2026-03-07T06:46:16Z)
• Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation [55.938648534942665]
  Test-time policy adaptation for multi-turn interactions (T2PAM) is essential for aligning Large Language Models (LLMs) with dynamic user needs during inference time.<n>We propose ROSA2, a framework that reformulates interaction as a joint optimization problem over the heterogeneous space of Words and Weights.
  arXiv  Detail & Related papers  (2026-03-02T02:16:20Z)
• Targeted Testing of Compiler Optimizations via Grammar-Level Composition Styles [8.598686284546773]
  Existing fuzzers struggle to test compiler optimizations effectively.<n>We propose targeted fuzzing of individual optimizations to complement pipeline-based testing.<n>Our evaluation on LLVM and MLIR shows that TargetFuzz improves coverage by 8% and 11%.
  arXiv  Detail & Related papers  (2025-12-04T00:13:25Z)
• Hybrid Fuzzing with LLM-Guided Input Mutation and Semantic Feedback [0.0]
  I present a hybrid fuzzing framework that integrates static and dynamic analysis with Large Language Model (LLM)-guided input mutation and semantic feedback.<n>Our method achieves faster time-to-first-bug, higher semantic diversity, and a competitive number of unique bugs compared to state-of-the-art fuzzers.
  arXiv  Detail & Related papers  (2025-11-06T02:38:24Z)
• Semantic-Aware Fuzzing: An Empirical Framework for LLM-Guided, Reasoning-Driven Input Mutation [0.5336076422485075]
  Security vulnerabilities in Internet-of-Things devices, mobile platforms, and autonomous systems remain critical.<n>Traditional mutation-based fuzzers primarily perform byte or bit-level edits without semantic reasoning.<n>We present an open-source framework that integrates reasoning LLMs with AFL++ on Google's FuzzBench.
  arXiv  Detail & Related papers  (2025-09-23T19:57:29Z)
• Boosting Skeleton-Driven SMT Solver Fuzzing by Leveraging LLM to Produce Formula Generators [5.527936960933817]
  Satisfiability Modulo Theory (SMT) solvers are foundational to modern systems and programming languages research.<n>While prior testing techniques performed well on earlier solver versions, they struggle to keep pace with rapidly evolving features.<n>Recent approaches based on Large Language Models (LLMs) show promise in exploring advanced solver capabilities.
  arXiv  Detail & Related papers  (2025-08-28T01:21:26Z)
• Mut4All: Fuzzing Compilers via LLM-Synthesized Mutators Learned from Bug Reports [35.036499451862355]
  Mutation-based fuzzing is effective for uncovering compiler bugs, but designing high-quality mutators for modern languages remains challenging.<n>We present Mut4All, a fully automated, language-agnostic framework that synthesizes mutators using Large Language Models (LLMs) and compiler-specific knowledge from bug reports.
  arXiv  Detail & Related papers  (2025-07-25T13:54:42Z)
• FLEXITOKENS: Flexible Tokenization for Evolving Language Models [3.2749495104311874]
  Language models (LMs) are challenging to adapt to new data distributions by simple finetuning.<n>This is due to the rigidity of their subword tokenizers, which typically remain unchanged during adaptation.<n>We develop byte-level LMs with learnable tokenizers to make tokenization adaptive.
  arXiv  Detail & Related papers  (2025-07-17T01:55:41Z)
• LLAMA: Multi-Feedback Smart Contract Fuzzing Framework with LLM-Guided Seed Generation [56.84049855266145]
  We propose a Multi-feedback Smart Contract Fuzzing framework (LLAMA) that integrates evolutionary mutation strategies, and hybrid testing techniques.<n>LLAMA achieves 91% instruction coverage and 90% branch coverage, while detecting 132 out of 148 known vulnerabilities.<n>These results highlight LLAMA's effectiveness, adaptability, and practicality in real-world smart contract security testing scenarios.
  arXiv  Detail & Related papers  (2025-07-16T09:46:58Z)
• Evaluating Sparse Autoencoders: From Shallow Design to Matching Pursuit [16.996218963146788]
  Sparse autoencoders (SAEs) have recently become central tools for interpretability.<n>This paper evaluates SAEs in a controlled setting using MNIST.<n>We introduce a multi-iteration SAE by unrolling Matching Pursuit (MP-SAE)
  arXiv  Detail & Related papers  (2025-06-05T16:57:58Z)
• Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective [50.261681681643076]
  We propose a novel metric called SemVarEffect and a benchmark named SemVarBench to evaluate the causality between semantic variations in inputs and outputs in text-to-image synthesis. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
  arXiv  Detail & Related papers  (2024-10-14T08:45:35Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [91.981439746404]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation. We first demonstrate the effectiveness of the QASemConsistency methodology for human annotation. We then implement several methods for automatically detecting localized factual inconsistencies.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• Training Language Models on Synthetic Edit Sequences Improves Code Synthesis [33.13471417703669]
  Large language models (LLMs) autoresourcedly synthesize programs in a single pass. We develop a synthetic data generation algorithm called LintSeq to generate high-quality code edit data. We show that edit sequence finetuned models produce more diverse programs than baselines synthesis.
  arXiv  Detail & Related papers  (2024-10-03T17:57:22Z)
• Fix the Tests: Augmenting LLMs to Repair Test Cases with Static Collector and Neural Reranker [9.428021853841296]
  We propose SYNTER, a novel approach to automatically repair obsolete test cases via precise and concise TROCtxs construction. With the augmentation of constructed TROCtxs, hallucinations are reduced by 57.1%.
  arXiv  Detail & Related papers  (2024-07-04T04:24:43Z)
• Nearest Neighbor Speculative Decoding for LLM Generation and Attribution [87.3259169631789]
  Nearest Speculative Decoding (NEST) is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources. NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks. In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B.
  arXiv  Detail & Related papers  (2024-05-29T17:55:03Z)
• SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation [55.2480439325792]
  We study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance.
  arXiv  Detail & Related papers  (2024-05-16T12:22:41Z)
• LLMorpheus: Mutation Testing using Large Language Models [7.312170216336085]
  This paper presents a technique where a Large Language Model (LLM) is prompted to suggest mutations by asking it what placeholders that have been inserted in source code could be replaced with. We find LLMorpheus to be capable of producing mutants that resemble existing bugs that cannot be produced by StrykerJS, a state-of-the-art mutation testing tool.
  arXiv  Detail & Related papers  (2024-04-15T17:25:14Z)
• MRL Parsing Without Tears: The Case of Hebrew [14.104766026682384]
  In morphologically rich languages (MRLs), wheres need to identify multiple lexical units in each token, existing systems suffer in latency and setup complexity. We present a new "flipped pipeline": decisions are made directly on the whole-token units by expert classifiers, each one dedicated to one specific task. This blazingly fast approach sets a new SOTA in Hebrew POS tagging and dependency parsing, while also reaching near-SOTA performance on other Hebrew tasks.
  arXiv  Detail & Related papers  (2024-03-11T17:54:33Z)
• A Novel Refactoring and Semantic Aware Abstract Syntax Tree Differencing Tool and a Benchmark for Evaluating the Accuracy of Diff Tools [2.0625936401496237]
  Syntax Tree (AST) diff tools were developed to overcome the limitations of line-based diff tools, which are used by the majority of developers. We propose a novel AST diff tool based on RefactoringMiner that resolves all aforementioned limitations. Our tool achieved a considerably higher precision and recall, especially for commits, with an execution time that is comparable with incompatible tools.
  arXiv  Detail & Related papers  (2024-03-09T15:32:41Z)
• Contrastive Instruction Tuning [61.97704869248903]
  We propose Contrastive Instruction Tuning to maximize the similarity between semantically equivalent instruction-instance pairs. Experiments on the PromptBench benchmark show that CoIN consistently improves LLMs' robustness to unseen instructions with variations across character, word, sentence, and semantic levels by an average of +2.5% in accuracy.
  arXiv  Detail & Related papers  (2024-02-17T00:09:32Z)
• Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition [62.83832841523525]
  We propose a fast and accurate parallel transformer, termed Paraformer. It accurately predicts the number of output tokens and extract hidden variables. It can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
  arXiv  Detail & Related papers  (2022-06-16T17:24:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.