Unboxing Diffusion Models for the Arts: Interactive Model Bending and Practice-Based Explainability

• URL: http://arxiv.org/abs/2607.22428v1
• Date: Fri, 24 Jul 2026 15:48:38 GMT
• Title: Unboxing Diffusion Models for the Arts: Interactive Model Bending and Practice-Based Explainability
• Abstract summary: We argue that even large models can function as creative materials when their internal structure is made visible and manipulable.<n>We instantiate this approach with a model bending and an interactive (inspection) interface integrated into ComfyUIs nodebased workflow.
• Score: 2.7138043802067835
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Explainable AI (XAI) in creative practice can be less about technocentric explanation and more about enabling artists to inspect modify and debug models as part of making Yet largescale texttoimage diffusion systems are typically presented as opaque endtoend tools limiting this kind of material engagement We argue that even large models can function as creative materials when their internal structure is made visible and manipulable To support this we propose a handson approach to explainability centred on experimentation and intervention We instantiate this approach with a model bending and an interactive (inspection) interface integrated into ComfyUIs nodebased workflow including interactive layer selection and intervention controls Through qualitative and quantitative analysis of bending interventions in Stable Diffusion 15 we show how manipulating specific components of a diffusion pipeline produces relatively consistent families of visual effects allowing artists to build practical layerlevel intuition about how different parts of the model shape generated images

Related papers

• Guava: An Effective and Universal Harness for Embodied Manipulation [74.34187069605844]
  We develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model.<n>Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation.
  arXiv  Detail & Related papers  (2026-06-16T18:09:26Z)
• JODA: Composable Joint Dynamics for Articulated Objects [17.60725863703415]
  JODA is a framework for generating joint-level dynamics as a structured three-channel field over the joint degree of freedom.<n>It defines a compact and expressive function space that is both interpretable and compatible with differentiable simulation.<n>We show that JODA enables plausible and controllable modeling of diverse joint behaviors, providing a unified interface for inference, editing, and optimization.
  arXiv  Detail & Related papers  (2026-05-11T04:02:15Z)
• Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting [60.15416769662556]
  We study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples.<n>We propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR)<n>We show how the learned primitives can be made controllable through a Bézier-based conditioning module.
  arXiv  Detail & Related papers  (2026-03-01T13:42:35Z)
• Explainability-in-Action: Enabling Expressive Manipulation and Tacit Understanding by Bending Diffusion Models in ComfyUI [6.349140286855134]
  We propose a craft-based approach to explainability rooted in long-term, hands-on engagement akin to Sch"on's "reflection-in-action"<n>We demonstrate that by interactively manipulating different parts of a generative model, artists can develop an intuition about how each component influences the output.
  arXiv  Detail & Related papers  (2025-08-10T05:19:30Z)
• JADE: Joint-aware Latent Diffusion for 3D Human Generative Modeling [62.77347895550087]
  We introduce JADE, a generative framework that learns the variations of human shapes with fined-grained control.<n>Our key insight is a joint-aware latent representation that decomposes human bodies into skeleton structures.<n>To generate coherent and plausible human shapes under our proposed decomposition, we also present a cascaded pipeline.
  arXiv  Detail & Related papers  (2024-12-29T14:18:35Z)
• Interaction as Explanation: A User Interaction-based Method for Explaining Image Classification Models [1.3597551064547502]
  In computer vision, explainable AI (xAI) methods seek to mitigate the 'black-box' problem. Traditional xAI methods concentrate on visualizing input features that influence model predictions. We present an interaction-based xAI method that enhances user comprehension of image classification models through their interaction.
  arXiv  Detail & Related papers  (2024-04-15T14:26:00Z)
• Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models [51.21351775178525]
  DiffExplainer is a novel framework that, leveraging language-vision models, enables multimodal global explainability. It employs diffusion models conditioned on optimized text prompts, synthesizing images that maximize class outputs. The analysis of generated visual descriptions allows for automatic identification of biases and spurious features.
  arXiv  Detail & Related papers  (2024-04-03T10:11:22Z)
• Neural Additive Image Model: Interpretation through Interpolation [0.0]
  We propose a holistic modeling approach utilizing Neural Additive Models and Diffusion Autoencoders. We demonstrate that the proposed method can precisely identify complex image effects in an ablation study. To further showcase the practical applicability of our proposed model, we conduct a case study in which we investigate how the distinctive features and attributes captured within host images exert influence on the pricing of Airbnb rentals.
  arXiv  Detail & Related papers  (2024-03-06T16:46:07Z)
• StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data [129.92449761766025]
  We propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models. Our research includes comprehensive experiments conducted on various datasets.
  arXiv  Detail & Related papers  (2023-08-20T12:43:52Z)
• Single Stage Virtual Try-on via Deformable Attention Flows [51.70606454288168]
  Virtual try-on aims to generate a photo-realistic fitting result given an in-shop garment and a reference person image. We develop a novel Deformable Attention Flow (DAFlow) which applies the deformable attention scheme to multi-flow estimation. Our proposed method achieves state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-07-19T10:01:31Z)
• Compositional Visual Generation with Composable Diffusion Models [80.75258849913574]
  We propose an alternative structured approach for compositional generation using diffusion models. An image is generated by composing a set of diffusion models, with each of them modeling a certain component of the image. The proposed method can generate scenes at test time that are substantially more complex than those seen in training.
  arXiv  Detail & Related papers  (2022-06-03T17:47:04Z)
• Structure-Regularized Attention for Deformable Object Representation [17.120035855774344]
  Capturing contextual dependencies has proven useful to improve the representational power of deep neural networks. Recent approaches that focus on modeling global context, such as self-attention and non-local operation, achieve this goal by enabling unconstrained pairwise interactions between elements. We consider learning representations for deformable objects which can benefit from context exploitation by modeling the structural dependencies that the data intrinsically possesses.
  arXiv  Detail & Related papers  (2021-06-12T03:10:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.