AnomalyXFusion: Multi-modal Anomaly Synthesis with Diffusion

• URL: http://arxiv.org/abs/2404.19444v2
• Date: Thu, 2 May 2024 01:12:02 GMT
• Title: AnomalyXFusion: Multi-modal Anomaly Synthesis with Diffusion
• Authors: Jie Hu, Yawen Huang, Yilin Lu, Guoyang Xie, Guannan Jiang, Yefeng Zheng, Zhichao Lu,
• Abstract summary: Anomaly synthesis is one of the effective methods to augment abnormal samples for training. We present the AnomalyXFusion framework, designed to harness multi-modality information to enhance the quality of synthesized abnormal samples.
• Score: 31.338732251924103
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Anomaly synthesis is one of the effective methods to augment abnormal samples for training. However, current anomaly synthesis methods predominantly rely on texture information as input, which limits the fidelity of synthesized abnormal samples. Because texture information is insufficient to correctly depict the pattern of anomalies, especially for logical anomalies. To surmount this obstacle, we present the AnomalyXFusion framework, designed to harness multi-modality information to enhance the quality of synthesized abnormal samples. The AnomalyXFusion framework comprises two distinct yet synergistic modules: the Multi-modal In-Fusion (MIF) module and the Dynamic Dif-Fusion (DDF) module. The MIF module refines modality alignment by aggregating and integrating various modality features into a unified embedding space, termed X-embedding, which includes image, text, and mask features. Concurrently, the DDF module facilitates controlled generation through an adaptive adjustment of X-embedding conditioned on the diffusion steps. In addition, to reveal the multi-modality representational power of AnomalyXFusion, we propose a new dataset, called MVTec Caption. More precisely, MVTec Caption extends 2.2k accurate image-mask-text annotations for the MVTec AD and LOCO datasets. Comprehensive evaluations demonstrate the effectiveness of AnomalyXFusion, especially regarding the fidelity and diversity for logical anomalies. Project page: http:github.com/hujiecpp/MVTec-Caption

Related papers

• LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model [77.66516875262963]
  We present textbfLLaDA-o, an effective and length-adaptive omni diffusion model for multimodal understanding and generation.<n>Building on MoD, we introduce a data-centric length adaptation strategy that enables flexible-length decoding in multimodal settings.<n>Experiments show that LLaDA-o achieves state-of-the-art performance among omni-diffusion models on multimodal understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2026-03-01T12:05:06Z)
• Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach [99.80480649258557]
  DiTFuse is an instruction-driven framework that performs semantics-aware fusion within a single model.<n>Experiments on public IVIF, MFF, and MEF benchmarks confirm superior quantitative and qualitative performance, sharper textures, and better semantic retention.
  arXiv  Detail & Related papers  (2025-12-08T05:04:54Z)
• Double Helix Diffusion for Cross-Domain Anomaly Image Generation [47.093354259479234]
  This paper introduces Double Helix Diffusion (DH-Diff), a novel cross-domain generative framework designed to simultaneously synthesize high-fidelity anomaly images and their pixel-level annotation masks.<n>DH-Diff employs a unique architecture inspired by a double helix, cycling through distinct modules for feature separation, connection, and merging.<n>Extensive experiments demonstrate that DH-Diff significantly outperforms state-of-the-art methods in diversity and authenticity, leading to significant improvements in downstream anomaly detection performance.
  arXiv  Detail & Related papers  (2025-09-16T08:06:07Z)
• Pyramid Hierarchical Masked Diffusion Model for Imaging Synthesis [6.475175425060296]
  The paper presents a novel image synthesis network, the Pyramid Hierarchical Masked Diffusion Model (PHMDiff)<n>Experiments on two challenging datasets demonstrate that PHMDiff achieves superior performance in both the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM)<n>The PHMDiff model, a multi-scale image synthesis framework across and within medical imaging modalities, shows significant advantages over other methods.
  arXiv  Detail & Related papers  (2025-07-22T13:30:54Z)
• Generate Aligned Anomaly: Region-Guided Few-Shot Anomaly Image-Mask Pair Synthesis for Industrial Inspection [53.137651284042434]
  Anomaly inspection plays a vital role in industrial manufacturing, but the scarcity of anomaly samples limits the effectiveness of existing methods.<n>We propose Generate grained Anomaly (GAA), a region-guided, few-shot anomaly image-mask pair generation framework.<n>GAA generates realistic, diverse, and semantically aligned anomalies using only a small number of samples.
  arXiv  Detail & Related papers  (2025-07-13T12:56:59Z)
• AMF-MedIT: An Efficient Align-Modulation-Fusion Framework for Medical Image-Tabular Data [7.265406812965239]
  AMF-MedIT is an efficient Align-Modulation-Fusion framework for medical image and tabular data integration.<n>AMF-MedIT achieves a superior balance between multimodal performance and data efficiency.
  arXiv  Detail & Related papers  (2025-06-24T09:10:35Z)
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces [10.85468238780625]
  We propose a novel framework for building multimodal diffusion models on arbitrary state spaces.<n>By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously.
  arXiv  Detail & Related papers  (2025-06-09T16:20:20Z)
• SynergyAmodal: Deocclude Anything with Text Control [27.027748040959025]
  Image deocclusion aims to recover the invisible regions (ie, shape and appearance) of occluded instances in images. We propose SynergyAmodal, a novel framework for co-synthesizing in-the-wild amodal datasets with comprehensive shape and appearance annotations.
  arXiv  Detail & Related papers  (2025-04-28T06:04:17Z)
• FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion [63.87313550399871]
  Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability. We propose Self-supervised Transfer (PST) and FrequencyDe-coupled Fusion module (FreDF) PST establishes cross-modal knowledge transfer through latent space alignment with image foundation models. FreDF explicitly decouples high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches.
  arXiv  Detail & Related papers  (2025-03-25T15:04:53Z)
• AnomalyControl: Learning Cross-modal Semantic Features for Controllable Anomaly Synthesis [9.659449396370023]
  We propose a novel anomaly synthesis framework called AnomalyControl to learn cross-modal semantic features as guidance signals. AnomalyControl can achieve state-of-the-art results in anomaly synthesis compared with existing methods.
  arXiv  Detail & Related papers  (2024-12-09T14:13:21Z)
• Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation [54.96563068182733]
  We propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task. MADM utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. We show that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities.
  arXiv  Detail & Related papers  (2024-10-29T03:49:40Z)
• Aggregation of Multi Diffusion Models for Enhancing Learned Representations [4.126721111013567]
  This paper introduces a novel algorithm, Aggregation of Multi Diffusion Models (AMDM) AMDM synthesizes features from multiple diffusion models into a specified model, enhancing its learned representations to activate specific features for fine-grained control. Experimental results demonstrate that AMDM significantly improves fine-grained control without additional training or inference time.
  arXiv  Detail & Related papers  (2024-10-02T06:16:06Z)
• Modality Prompts for Arbitrary Modality Salient Object Detection [57.610000247519196]
  This paper delves into the task of arbitrary modality salient object detection (AM SOD) It aims to detect salient objects from arbitrary modalities, eg RGB images, RGB-D images, and RGB-D-T images. A novel modality-adaptive Transformer (MAT) will be proposed to investigate two fundamental challenges of AM SOD.
  arXiv  Detail & Related papers  (2024-05-06T11:02:02Z)
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion [144.9653045465908]
  We propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM) Our approach yields promising fusion results in infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2023-03-13T04:06:42Z)
• One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale [36.590918776922905]
  This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model.
  arXiv  Detail & Related papers  (2023-03-12T03:38:39Z)
• TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis [6.899641625551976]
  We introduce a pure transformer-based method, which we refer to as Throughout Fusion Transformer (TFormer)", for sufficient information intergration in MSLD. We then carefully design a stack of dual-branch hierarchical multi-modal transformer (HMT) blocks to fuse information across different image modalities in a stage-by-stage way. Our TFormer outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-11-21T12:07:05Z)
• f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation [56.04628143914542]
  Diffusion models (DMs) have recently emerged as SoTA tools for generative modeling in various domains. We propose f-DM, a generalized family of DMs which allows progressive signal transformation. We apply f-DM in image generation tasks with a range of functions, including down-sampling, blurring, and learned transformations.
  arXiv  Detail & Related papers  (2022-10-10T18:49:25Z)
• A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion [54.512440195060584]
  We propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM) UMM-CSGM employs a novel multi-in multi-out Conditional Score Network (mm-CSN) to learn a comprehensive set of cross-modal conditional distributions. Experiments on BraTS19 dataset show that the UMM-CSGM can more reliably synthesize the heterogeneous enhancement and irregular area in tumor-induced lesions.
  arXiv  Detail & Related papers  (2022-07-07T16:57:21Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.