BlobCtrl: A Unified and Flexible Framework for Element-level Image Generation and Editing

• URL: http://arxiv.org/abs/2503.13434v1
• Date: Mon, 17 Mar 2025 17:58:05 GMT
• Title: BlobCtrl: A Unified and Flexible Framework for Element-level Image Generation and Editing
• Authors: Yaowei Li, Lingen Li, Zhaoyang Zhang, Xiaoyu Li, Guangzhi Wang, Hongxiang Li, Xiaodong Cun, Ying Shan, Yuexian Zou,
• Abstract summary: BlobCtrl is a framework that unifies element-level generation and editing using a probabilistic blob-based representation.<n>Our approach effectively decouples and represents spatial location, semantic content, and identity information.<n> Experiments show that BlobCtrl excels in various element-level manipulation tasks while maintaining computational efficiency.
• Score: 86.26405009039868
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Element-level visual manipulation is essential in digital content creation, but current diffusion-based methods lack the precision and flexibility of traditional tools. In this work, we introduce BlobCtrl, a framework that unifies element-level generation and editing using a probabilistic blob-based representation. By employing blobs as visual primitives, our approach effectively decouples and represents spatial location, semantic content, and identity information, enabling precise element-level manipulation. Our key contributions include: 1) a dual-branch diffusion architecture with hierarchical feature fusion for seamless foreground-background integration; 2) a self-supervised training paradigm with tailored data augmentation and score functions; and 3) controllable dropout strategies to balance fidelity and diversity. To support further research, we introduce BlobData for large-scale training and BlobBench for systematic evaluation. Experiments show that BlobCtrl excels in various element-level manipulation tasks while maintaining computational efficiency, offering a practical solution for precise and flexible visual content creation. Project page: https://liyaowei-stu.github.io/project/BlobCtrl/

Related papers

• BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations [82.94002870060045]
  Existing video generation models struggle to follow complex text prompts and synthesize multiple objects.<n>We develop a blob-grounded video diffusion model named BlobGEN-Vid that allows users to control object motions and fine-grained object appearance.<n>We show that our framework is model-agnostic and build BlobGEN-Vid based on both U-Net and DiT-based video diffusion models.
  arXiv  Detail & Related papers  (2025-01-13T19:17:06Z)
• A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap [50.079224604394]
  We present a novel model-agnostic framework called textbfContext-textbfEnhanced textbfFeature textbfAment (CEFA) CEFA consists of a feature alignment module and a context enhancement module. Our method can serve as a plug-and-play module to improve the detection performance of HOI models on rare categories.
  arXiv  Detail & Related papers  (2024-07-31T08:42:48Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets. We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• Deep Structure and Attention Aware Subspace Clustering [29.967881186297582]
  We propose a novel Deep Structure and Attention aware Subspace Clustering (DSASC) We use a vision transformer to extract features, and the extracted features are divided into two parts, structure features, and content features. Our method significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-25T01:19:47Z)
• Optimal Transport Aggregation for Visual Place Recognition [9.192660643226372]
  We introduce SALAD, which reformulates NetVLAD's soft-assignment of local features to clusters as an optimal transport problem. In SALAD, we consider both feature-to-cluster and cluster-to-feature relations and we also introduce a 'dustbin' cluster, designed to selectively discard features deemed non-informative. Our single-stage method surpasses single-stage baselines in public VPR datasets, but also surpasses two-stage methods that add a re-ranking with significantly higher cost.
  arXiv  Detail & Related papers  (2023-11-27T15:46:19Z)
• ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer [91.43066633305662]
  We propose a novel underlineComPlementary underlinetransformer, textbfComPtr, for diverse bi-source dense prediction tasks. ComPtr treats different inputs equally and builds an efficient dense interaction model in the form of sequence-to-sequence on top of the transformer.
  arXiv  Detail & Related papers  (2023-07-23T15:17:45Z)
• Multi-level Second-order Few-shot Learning [111.0648869396828]
  We propose a Multi-level Second-order (MlSo) few-shot learning network for supervised or unsupervised few-shot image classification and few-shot action recognition. We leverage so-called power-normalized second-order base learner streams combined with features that express multiple levels of visual abstraction. We demonstrate respectable results on standard datasets such as Omniglot, mini-ImageNet, tiered-ImageNet, Open MIC, fine-grained datasets such as CUB Birds, Stanford Dogs and Cars, and action recognition datasets such as HMDB51, UCF101, and mini-MIT.
  arXiv  Detail & Related papers  (2022-01-15T19:49:00Z)
• Joint Self-Attention and Scale-Aggregation for Self-Calibrated Deraining Network [13.628218953897946]
  In this paper, we propose an effective algorithm, called JDNet, to solve the single image deraining problem. By designing the Scale-Aggregation and Self-Attention modules with Self-Calibrated convolution skillfully, the proposed model has better deraining results.
  arXiv  Detail & Related papers  (2020-08-06T17:04:34Z)
• Learning Semantically Enhanced Feature for Fine-Grained Image Classification [27.136912902584093]
  Our approach learns fine-grained features by enhancing the semantics of sub-features of a global feature. Our approach is parameter parsimonious and can be easily integrated into the backbone model as a plug-and-play module for end-to-end training.
  arXiv  Detail & Related papers  (2020-06-24T03:41:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.