Modular Diffusion Models for Structured Visual Recognition

• URL: http://arxiv.org/abs/2606.22702v1
• Date: Sun, 21 Jun 2026 22:38:35 GMT
• Title: Modular Diffusion Models for Structured Visual Recognition
• Abstract summary: We present Modular Diffusion Models (MDMs), a simple and novel framework that learns a distribution over structured outputs for a given input image.<n>MDMs decompose the diffusion process into distinct, task-specific modules, each focused on capturing a different aspect of the structured information space.<n>This modular design allows each component to be learned independently, with seamless integration at inference without additional training.
• Score: 52.38687800325397
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional supervised methods for structured visual recognition tasks -- such as object detection, segmentation, and scene graph generation -- often produce deterministic, fixed outputs, limiting their ability to capture the inherent uncertainty in complex visual scenes. As a consequence, such point estimates are unable to capture the prediction uncertainty (or multi modality) intrinsic to these problems, often arising from natural ambiguities (e.g., ambiguity in size of partially occluded objects, local ambiguity of exact segmentation boundary, etc.) as well as noise and sparsity of training data. To address this limitation, we present Modular Diffusion Models (MDMs), a simple and novel framework that learns a distribution over structured outputs for a given input image. MDMs decompose the diffusion process into distinct, task-specific modules, each focused on capturing a different aspect of the structured information space, such as object categories, spatial locations, and inter-object relationships. This modular design allows each component to be learned independently, with seamless integration at inference without additional training. Furthermore, the modularity of MDMs enables the diffusion process to easily operate over the heterogeneous output space common in many structured learning tasks (e.g., a continuous bounding boxes and discrete class labels). Experimental results over three distinct structured tasks -- object detection, instance segmentation, and scene graph generation -- highlight the benefits of our proposed framework.

Related papers

• Discrete Diffusion Models: A Unified Framework from Tokenization to Generation [66.16696467962375]
  This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space.<n>Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space.
  arXiv  Detail & Related papers  (2026-07-15T04:23:22Z)
• Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection [5.097809301149341]
  In real-world deployments, some modalities may be unavailable due to sensor failures, acquisition issues, or challenging atmospheric conditions.<n>Existing multimodal segmentation models typically address missing modalities by learning a shared representation across inputs.<n>We tackle this limitation with CBC-SLP, a multimodal semantic segmentation model designed to preserve both modality-invariant and modality-specific information.
  arXiv  Detail & Related papers  (2026-04-17T09:05:22Z)
• Unified modality separation: A vision-language framework for unsupervised domain adaptation [60.8391821117794]
  Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains.<n>We propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components.<n>Our methods achieve up to 9% performance gain with 9 times of computational efficiencies.
  arXiv  Detail & Related papers  (2025-08-07T02:51:10Z)
• Segment Concealed Objects with Incomplete Supervision [63.637733655439334]
  Incompletely-Supervised Concealed Object (ISCOS) involves segmenting objects that seamlessly blend into their surrounding environments.<n>This task remains highly challenging due to the limited supervision provided by the incompletely annotated training data.<n>In this paper, we introduce the first unified method for ISCOS to address these challenges.
  arXiv  Detail & Related papers  (2025-06-10T16:25:15Z)
• Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objects [13.311084447321234]
  This study introduces interpretable metrics quantifying object tree-likeness and textural separability.<n>On carefully controlled synthetic experiments and real-world datasets, we show that SFM performance noticeably correlates with these factors.<n>We link these failures to "textural confusion", where models misinterpret local structure as global texture, causing over-segmentation or difficulty distinguishing objects from similar backgrounds.
  arXiv  Detail & Related papers  (2024-12-05T15:25:51Z)
• Sequential Amodal Segmentation via Cumulative Occlusion Learning [15.729212571002906]
  A visual system must be able to segment both the visible and occluded regions of objects, while discerning their occlusion order. We introduce a diffusion model with cumulative occlusion learning designed for sequential amodal segmentation of objects with uncertain categories. This model iteratively refines the prediction using the cumulative mask strategy during diffusion, effectively capturing the uncertainty of invisible regions. It is akin to the human capability for amodal perception, i.e., to decipher the spatial ordering among objects and accurately predict complete contours for occluded objects in densely layered visual scenes.
  arXiv  Detail & Related papers  (2024-05-09T14:17:26Z)
• Support-set based Multi-modal Representation Enhancement for Video Captioning [121.70886789958799]
  We propose a Support-set based Multi-modal Representation Enhancement (SMRE) model to mine rich information in a semantic subspace shared between samples. Specifically, we propose a Support-set Construction (SC) module to construct a support-set to learn underlying connections between samples and obtain semantic-related visual elements. During this process, we design a Semantic Space Transformation (SST) module to constrain relative distance and administrate multi-modal interactions in a self-supervised way.
  arXiv  Detail & Related papers  (2022-05-19T03:40:29Z)
• Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization [98.46318529630109]
  We take inspiration from traditional spectral segmentation methods by reframing image decomposition as a graph partitioning problem. We find that these eigenvectors already decompose an image into meaningful segments, and can be readily used to localize objects in a scene. By clustering the features associated with these segments across a dataset, we can obtain well-delineated, nameable regions.
  arXiv  Detail & Related papers  (2022-05-16T17:47:44Z)
• CellSegmenter: unsupervised representation learning and instance segmentation of modular images [0.0]
  We introduce a structured deep generative model and an amortized inference framework for unsupervised representation learning and instance segmentation tasks. The proposed inference algorithm is convolutional and parallelized, without any recurrent mechanisms. We show segmentation results obtained for a cell nuclei imaging dataset, demonstrating the ability of our method to provide high-quality segmentations.
  arXiv  Detail & Related papers  (2020-11-25T02:10:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.