Explainability-in-Action: Enabling Expressive Manipulation and Tacit Understanding by Bending Diffusion Models in ComfyUI

• URL: http://arxiv.org/abs/2508.07183v1
• Date: Sun, 10 Aug 2025 05:19:30 GMT
• Title: Explainability-in-Action: Enabling Expressive Manipulation and Tacit Understanding by Bending Diffusion Models in ComfyUI
• Abstract summary: We propose a craft-based approach to explainability rooted in long-term, hands-on engagement akin to Sch"on's "reflection-in-action"<n>We demonstrate that by interactively manipulating different parts of a generative model, artists can develop an intuition about how each component influences the output.
• Score: 6.349140286855134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explainable AI (XAI) in creative contexts can go beyond transparency to support artistic engagement, modifiability, and sustained practice. While curated datasets and training human-scale models can offer artists greater agency and control, large-scale generative models like text-to-image diffusion systems often obscure these possibilities. We suggest that even large models can be treated as creative materials if their internal structure is exposed and manipulable. We propose a craft-based approach to explainability rooted in long-term, hands-on engagement akin to Sch\"on's "reflection-in-action" and demonstrate its application through a model-bending and inspection plugin integrated into the node-based interface of ComfyUI. We demonstrate that by interactively manipulating different parts of a generative model, artists can develop an intuition about how each component influences the output.

Related papers

• Unboxing Diffusion Models for the Arts: Interactive Model Bending and Practice-Based Explainability [2.7138043802067835]
  We argue that even large models can function as creative materials when their internal structure is made visible and manipulable.<n>We instantiate this approach with a model bending and an interactive (inspection) interface integrated into ComfyUIs nodebased workflow.
  arXiv  Detail & Related papers  (2026-07-24T15:48:38Z)
• Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting [60.15416769662556]
  We study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples.<n>We propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR)<n>We show how the learned primitives can be made controllable through a Bézier-based conditioning module.
  arXiv  Detail & Related papers  (2026-03-01T13:42:35Z)
• Training Data Attribution for Image Generation using Ontology-Aligned Knowledge Graphs [3.686386213696443]
  We introduce a framework for interpreting generative outputs through the automatic construction of knowledge graphs.<n>Our method extracts structured triples from images, aligned with a domain-specific ontology.<n>By comparing the KGs of generated and training images, we can trace potential influences, enabling copyright analysis, dataset transparency, and interpretable AI.
  arXiv  Detail & Related papers  (2025-12-02T12:45:20Z)
• Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset [113.25650486482762]
  We introduce the Seamless Interaction dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage.<n>This dataset enables the development of AI technologies that understand dyadic embodied dynamics.<n>We develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech.
  arXiv  Detail & Related papers  (2025-06-27T18:09:49Z)
• ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing [25.610375901522886]
  ArtCrafter is a novel framework for text-to-image style transfer.<n>We introduce an attention-based style extraction module.<n>We also present a novel text-image aligning augmentation component.
  arXiv  Detail & Related papers  (2025-01-03T19:17:27Z)
• VisioBlend: Sketch and Stroke-Guided Denoising Diffusion Probabilistic Model for Realistic Image Generation [0.0]
  We propose a unified framework supporting three-dimensional control over image synthesis from sketches and strokes based on diffusion models. It enables users to decide the level of faithfulness to the input strokes and sketches. It solves the problem of data availability by synthesizing new data points from hand-drawn sketches and strokes.
  arXiv  Detail & Related papers  (2024-05-15T11:27:27Z)
• Interaction as Explanation: A User Interaction-based Method for Explaining Image Classification Models [1.3597551064547502]
  In computer vision, explainable AI (xAI) methods seek to mitigate the 'black-box' problem. Traditional xAI methods concentrate on visualizing input features that influence model predictions. We present an interaction-based xAI method that enhances user comprehension of image classification models through their interaction.
  arXiv  Detail & Related papers  (2024-04-15T14:26:00Z)
• Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models [51.21351775178525]
  DiffExplainer is a novel framework that, leveraging language-vision models, enables multimodal global explainability. It employs diffusion models conditioned on optimized text prompts, synthesizing images that maximize class outputs. The analysis of generated visual descriptions allows for automatic identification of biases and spurious features.
  arXiv  Detail & Related papers  (2024-04-03T10:11:22Z)
• Choreographing the Digital Canvas: A Machine Learning Approach to Artistic Performance [9.218587190403174]
  This paper introduces the concept of a design tool for artistic performances based on attribute descriptions. The platform integrates a novel machine-learning (ML) model with an interactive interface to generate and visualize artistic movements.
  arXiv  Detail & Related papers  (2024-03-26T01:42:13Z)
• CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion [73.08710648258985]
  Key painting attributes including layout, perspective, shape, and semantics often cannot be conveyed and expressed through style transfer.<n>Large-scale pretrained text-to-image generation models have demonstrated their capability to synthesize a vast amount of high-quality images.<n>Our main novel idea is to integrate multimodal semantic information as a synthesis guide into artworks, rather than transferring style to the real world.
  arXiv  Detail & Related papers  (2024-01-25T10:42:09Z)
• SODA: Bottleneck Diffusion Models for Representation Learning [75.7331354734152]
  We introduce SODA, a self-supervised diffusion model, designed for representation learning. The model incorporates an image encoder, which distills a source view into a compact representation, that guides the generation of related novel views. We show that by imposing a tight bottleneck between the encoder and a denoising decoder, we can turn diffusion models into strong representation learners.
  arXiv  Detail & Related papers  (2023-11-29T18:53:34Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.