MaTe: Images Are All You Need for Material Transfer via Diffusion Transformer

• URL: http://arxiv.org/abs/2605.15660v1
• Date: Fri, 15 May 2026 06:31:41 GMT
• Title: MaTe: Images Are All You Need for Material Transfer via Diffusion Transformer
• Abstract summary: MaTe is a streamlined diffusion framework that eliminates textual guidance and reference networks.<n>It integrates input images at the token level, enabling unified processing via multi-modal attention in a shared latent space.<n>It outperforms state-of-the-art methods in both visual quality and efficiency while preserving precise detail alignment.
• Score: 21.547583825573255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent diffusion-based methods for material transfer rely on image fine-tuning or complex architectures with assistive networks, but face challenges including text dependency, extra computational costs, and feature misalignment. To address these limitations, we propose MaTe, a streamlined diffusion framework that eliminates textual guidance and reference networks. MaTe integrates input images at the token level, enabling unified processing via multi-modal attention in a shared latent space. This design removes the need for additional adapters, ControlNet, inversion sampling, or model fine-tuning. Extensive experiments demonstrate that MaTe achieves high-quality material generation under a zero-shot, training-free paradigm. It outperforms state-of-the-art methods in both visual quality and efficiency while preserving precise detail alignment, significantly simplifying inference prerequisites.

Related papers

• Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction [21.45273679723765]
  Adverse weather causes significant image degradation, disrupting feature representation.<n>We propose a mask-guided MMIF method that integrates feature restoration and interaction.<n>Our method surpasses state-of-the-art approaches in visual quality, quantitative metrics, and downstream tasks.
  arXiv  Detail & Related papers  (2026-06-25T09:54:32Z)
• DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer [45.232470509013815]
  DealMaTe is a diffusion framework that eliminates text guidance and reference networks.<n>We show that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials.
  arXiv  Detail & Related papers  (2026-05-15T07:06:39Z)
• Multimodal Large Language Models for Multi-Subject In-Context Image Generation [56.20395856287325]
  Music is the first MLLM specifically designed for textbfMUlti-textbfSubject textbfIn-textbfContext image generation.<n>We enhance the model's understanding of multi-subject semantic relationships through a vision chain-of-thought mechanism.<n>By incorporating complex subject images during training, we improve the model's capacity for chained reasoning.
  arXiv  Detail & Related papers  (2026-04-08T15:37:42Z)
• Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach [99.80480649258557]
  DiTFuse is an instruction-driven framework that performs semantics-aware fusion within a single model.<n>Experiments on public IVIF, MFF, and MEF benchmarks confirm superior quantitative and qualitative performance, sharper textures, and better semantic retention.
  arXiv  Detail & Related papers  (2025-12-08T05:04:54Z)
• d-Sketch: Improving Visual Fidelity of Sketch-to-Image Translation with Pretrained Latent Diffusion Models without Retraining [18.73832646369506]
  We introduce a technique for sketch-to-image translation by exploiting the feature generalization capabilities of a large-scale diffusion model without retraining.<n> Experimental results demonstrate that the proposed method outperforms the existing techniques in qualitative and quantitative benchmarks.
  arXiv  Detail & Related papers  (2025-02-19T11:54:45Z)
• Stable Flow: Vital Layers for Training-Free Image Editing [74.52248787189302]
  Diffusion models have revolutionized the field of content synthesis and editing.<n>Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT)<n>We propose an automatic method to identify "vital layers" within DiT, crucial for image formation.<n>Next, to enable real-image editing, we introduce an improved image inversion method for flow models.
  arXiv  Detail & Related papers  (2024-11-21T18:59:51Z)
• Training-Free Layout-to-Image Generation with Marginal Attention Constraints [73.55660250459132]
  We propose a training-free layout-to-image (L2I) approach, which eliminates the need for additional modules or fine-tuning.<n>Specifically, we use text-visual cross-attention feature maps to quantify inconsistencies between the layout of the generated images and the provided instructions.<n>We leverage pixel-to-pixel correlations in the self-attention feature maps to align cross-attention maps and combine three loss functions constrained by boundary attention to update latent features.
  arXiv  Detail & Related papers  (2024-11-15T05:44:45Z)
• Training-free Composite Scene Generation for Layout-to-Image Synthesis [29.186425845897947]
  This paper introduces a novel training-free approach designed to overcome adversarial semantic intersections during the diffusion conditioning phase. We propose two innovative constraints: 1) an inter-token constraint that resolves token conflicts to ensure accurate concept synthesis; and 2) a self-attention constraint that improves pixel-to-pixel relationships. Our evaluations confirm the effectiveness of leveraging layout information for guiding the diffusion process, generating content-rich images with enhanced fidelity and complexity.
  arXiv  Detail & Related papers  (2024-07-18T15:48:07Z)
• TextCraftor: Your Text Encoder Can be Image Quality Controller [65.27457900325462]
  Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have revolutionized the field of content generation. We propose a proposed fine-tuning approach, TextCraftor, to enhance the performance of text-to-image diffusion models.
  arXiv  Detail & Related papers  (2024-03-27T19:52:55Z)
• Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis [15.76266032768078]
  This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries.<n>We first introduce vision guidance as a foundational spatial cue within the perturbed distribution.<n>We propose a universal framework, Layered Rendering Diffusion (LRDiff), which constructs an image-rendering process with multiple layers.
  arXiv  Detail & Related papers  (2023-11-30T10:36:19Z)
• Fine-grained Appearance Transfer with Diffusion Models [23.29713777525402]
  Image-to-image translation (I2I) seeks to alter the visual appearance between images while maintaining structural coherence. This paper proposes an innovative framework designed to surmount these challenges by integrating various aspects of semantic matching, appearance transfer, and latent deviation.
  arXiv  Detail & Related papers  (2023-11-27T04:00:04Z)
• MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask [84.84034179136458]
  A crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning. We propose an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models.
  arXiv  Detail & Related papers  (2023-09-08T15:53:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.