ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback

• URL: http://arxiv.org/abs/2505.17908v1
• Date: Fri, 23 May 2025 13:53:03 GMT
• Title: ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback
• Abstract summary: We present ComfyMind, a collaborative AI system designed to enable robust and scalable general-purpose generation.<n>ComfyMind introduces two core innovations: Semantic Interface (SWI) that abstracts low-level node graphs into callable functional language, and Search Tree Planning mechanism.<n>We evaluate ComfyMind on three public benchmarks: ComfyBench, GenEval, and Reason-Edit, which span generation, editing, and reasoning tasks.
• Score: 15.363560226232668
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid advancement of generative models, general-purpose generation has gained increasing attention as a promising approach to unify diverse tasks across modalities within a single system. Despite this progress, existing open-source frameworks often remain fragile and struggle to support complex real-world applications due to the lack of structured workflow planning and execution-level feedback. To address these limitations, we present ComfyMind, a collaborative AI system designed to enable robust and scalable general-purpose generation, built on the ComfyUI platform. ComfyMind introduces two core innovations: Semantic Workflow Interface (SWI) that abstracts low-level node graphs into callable functional modules described in natural language, enabling high-level composition and reducing structural errors; Search Tree Planning mechanism with localized feedback execution, which models generation as a hierarchical decision process and allows adaptive correction at each stage. Together, these components improve the stability and flexibility of complex generative workflows. We evaluate ComfyMind on three public benchmarks: ComfyBench, GenEval, and Reason-Edit, which span generation, editing, and reasoning tasks. Results show that ComfyMind consistently outperforms existing open-source baselines and achieves performance comparable to GPT-Image-1. ComfyMind paves a promising path for the development of open-source general-purpose generative AI systems. Project page: https://github.com/LitaoGuo/ComfyMind

Related papers

• Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learning [56.99010101756807]
  We propose Sync-R1, an end-to-end reinforcement learning framework that jointly optimize personalized understanding and generation.<n> Sync-R1 enables personalized comprehension to guide content creation, while the resulting generation quality reciprocally refines understanding.<n>We also introduce Dynamic Group Scaling (DGS), which adaptively filters low-potential trajectories to reduce gradient variance and accelerate convergence.<n> Experimental results demonstrate that Sync-R1 achieves state-of-the-art performance, showcasing superior cross-task reasoning and robust personalization.
  arXiv  Detail & Related papers  (2026-05-11T12:18:26Z)
• Architecture-Aware Multi-Design Generation for Repository-Level Feature Addition [53.50448142467294]
  RAIM is a multi-design and architecture-aware framework for repository-level feature addition.<n>It shifts away from linear patching by generating multiple diverse implementation designs.<n>Experiments on the NoCode-bench Verified dataset demonstrate that RAIM establishes a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-02T12:50:40Z)
• TodoEvolve: Learning to Architect Agent Planning Systems [68.48983335970901]
  TodoEvolve is a meta-planning paradigm that autonomously synthesizes and dynamically revises task-specific planning.<n>PlanFactory provides a common interface for heterogeneous planning patterns.<n>TodoEvolve consistently surpasses carefully engineered planning modules while maintaining economical API costs and runtime overhead.
  arXiv  Detail & Related papers  (2026-02-08T06:37:01Z)
• Embodied Task Planning via Graph-Informed Action Generation with Large Lanaguage Model [6.302973995986435]
  GiG is a novel planning framework that structures embodied agents' memory using a Graph-in-Graph architecture.<n>By clustering these graph embeddings, the framework enables retrieval of structure-aware priors.<n>We evaluate our framework on three embodied planning benchmarks-Robotouille Synchronous, Robotouille Asynchronous, and ALFWorld.
  arXiv  Detail & Related papers  (2026-01-29T15:18:58Z)
• VSA:Visual-Structural Alignment for UI-to-Code [29.15071743239679]
  We propose bfVSA (VSA), a multi-stage paradigm designed to synthesize organized assets through visual-text alignment.<n>Our framework yields a substantial improvement in code modularity and architectural consistency over state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2025-12-23T03:55:45Z)
• IACT: A Self-Organizing Recursive Model for General AI Agents: A Technical White Paper on the Architecture Behind kragent.ai [0.0]
  Interactive Agents Call Tree (IACT) is a general-purpose autonomous system driven purely by user dialogue.<n>We describe the architecture, design principles, and practical lessons behind the deployment of this model in the kragent.ai system.
  arXiv  Detail & Related papers  (2025-12-02T10:10:56Z)
• Dynamic Generation of Multi-LLM Agents Communication Topologies with Graph Diffusion Models [99.85131798240808]
  We introduce a novel generative framework called textitGuided Topology Diffusion (GTD)<n>Inspired by conditional discrete graph diffusion models, GTD formulates topology synthesis as an iterative construction process.<n>At each step, the generation is steered by a lightweight proxy model that predicts multi-objective rewards.<n>Experiments show that GTD can generate highly task-adaptive, sparse, and efficient communication topologies.
  arXiv  Detail & Related papers  (2025-10-09T05:28:28Z)
• JoyAgent-JDGenie: Technical Report on the GAIA [27.025464023889853]
  Large Language Models are increasingly deployed as autonomous agents for complex real-world tasks.<n>We propose a generalist agent architecture that integrates planning and execution agents with critic model voting, a hierarchical memory system spanning working, semantic, and procedural layers, and a refined tool suite for search, code execution, and multimodal parsing.
  arXiv  Detail & Related papers  (2025-10-01T04:41:58Z)
• Towards Efficient General Feature Prediction in Masked Skeleton Modeling [59.46799426434277]
  We propose a novel General Feature Prediction framework (GFP) for efficient mask skeleton modeling.<n>Our key innovation is replacing conventional low-level reconstruction with high-level feature prediction that spans from local motion patterns to global semantic representations.
  arXiv  Detail & Related papers  (2025-09-03T18:05:02Z)
• Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation [65.30763239365928]
  We introduce Genie Envisioner (GE), a unified world foundation platform for robotic manipulation.<n>GE integrates policy learning, evaluation, and simulation within a single video-generative framework.
  arXiv  Detail & Related papers  (2025-08-07T17:59:44Z)
• Assemble Your Crew: Automatic Multi-agent Communication Topology Design via Autoregressive Graph Generation [72.44384066166147]
  Multi-agent systems (MAS) based on large language models (LLMs) have emerged as a powerful solution for dealing with complex problems across diverse domains.<n>Existing approaches are fundamentally constrained by their reliance on a template graph modification paradigm with a predefined set of agents and hard-coded interaction structures.<n>We propose ARG-Designer, a novel autoregressive model that operationalizes this paradigm by constructing the collaboration graph from scratch.
  arXiv  Detail & Related papers  (2025-07-24T09:17:41Z)
• GenerationPrograms: Fine-grained Attribution with Executable Programs [72.23792263905372]
  We introduce a modular generation framework, GenerationPrograms, inspired by recent advancements in "code agent" architectures.<n>GenerationPrograms decomposes the process into two distinct stages: first, creating an executable program plan composed of modular text operations explicitly tailored to the query, and second, executing these operations following the program's specified instructions to produce the final response.<n> Empirical evaluations demonstrate that GenerationPrograms significantly improves attribution quality at both the document level and sentence level.
  arXiv  Detail & Related papers  (2025-06-17T14:37:09Z)
• Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation [7.980497203230983]
  We propose an Aggregation Structural Representation (ASR) module that integrates graph networks with large language models (LLMs) to preserve structural information while enhancing generative capability.<n>A comprehensive evaluation on the RICO dataset demonstrates the strong performance of ASR, both quantitatively using mean Intersection over Union (mIoU) and qualitatively through a crowdsourced user study.
  arXiv  Detail & Related papers  (2025-05-26T06:17:21Z)
• MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering [57.156093929365255]
  Gym-style framework for systematically reinforcement learning, evaluating, and improving autonomous large language model (LLM) agents.<n>MLE-Dojo covers diverse, open-ended MLE tasks carefully curated to reflect realistic engineering scenarios.<n>Its fully executable environment supports comprehensive agent training via both supervised fine-tuning and reinforcement learning.
  arXiv  Detail & Related papers  (2025-05-12T17:35:43Z)
• Flow State: Humans Enabling AI Systems to Program Themselves [0.24578723416255752]
  We introduce Pocketflow, a platform centered on Human-AI co-design.<n>Pocketflow is a Python framework built upon a deliberately minimal yet synergistic set of core abstractions.<n>It provides a robust, vendor-agnostic foundation with very little code that demonstrably reduces overhead.
  arXiv  Detail & Related papers  (2025-04-03T05:25:46Z)
• InvFussion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems [76.39776789410088]
  This work introduces a framework that combines the strong performance of supervised approaches and the flexibility of zero-shot methods.<n>A novel architectural design seamlessly integrates the degradation operator directly into the denoiser.<n> Experimental results on the FFHQ and ImageNet datasets demonstrate state-of-the-art posterior-sampling performance.
  arXiv  Detail & Related papers  (2025-04-02T12:40:57Z)
• RGL: A Graph-Centric, Modular Framework for Efficient Retrieval-Augmented Generation on Graphs [58.10503898336799]
  We introduce the RAG-on-Graphs Library (RGL), a modular framework that seamlessly integrates the complete RAG pipeline.<n>RGL addresses key challenges by supporting a variety of graph formats and integrating optimized implementations for essential components.<n>Our evaluations demonstrate that RGL not only accelerates the prototyping process but also enhances the performance and applicability of graph-based RAG systems.
  arXiv  Detail & Related papers  (2025-03-25T03:21:48Z)
• Multi-Objective Bayesian Optimization for Networked Black-Box Systems: A Path to Greener Profits and Smarter Designs [0.0]
  MOBONS is a novel Bayesian optimization-inspired algorithm that can efficiently optimize general function networks.<n>We demonstrate the effectiveness of MOBONS through two case studies, including one related to sustainable process design.
  arXiv  Detail & Related papers  (2025-02-19T21:49:05Z)
• ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems [80.69865295743149]
  This work attempts to study using LLM-based agents to design collaborative AI systems autonomously.<n>Based on ComfyBench, we develop ComfyAgent, a framework that empowers agents to autonomously design collaborative AI systems by generating.<n>While ComfyAgent achieves a comparable resolve rate to o1-preview and significantly surpasses other agents on ComfyBench, ComfyAgent has resolved only 15% of creative tasks.
  arXiv  Detail & Related papers  (2024-09-02T17:44:10Z)
• Effective Reinforcement Learning Based on Structural Information Principles [19.82391136775341]
  We propose a novel and general Structural Information principles-based framework for effective Decision-Making, namely SIDM. SIDM can be flexibly incorporated into various single-agent and multi-agent RL algorithms, enhancing their performance.
  arXiv  Detail & Related papers  (2024-04-15T13:02:00Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Bayesian Prompt Learning for Image-Language Model Generalization [64.50204877434878]
  We use the regularization ability of Bayesian methods to frame prompt learning as a variational inference problem. Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts. We demonstrate empirically on 15 benchmarks that Bayesian prompt learning provides an appropriate coverage of the prompt space.
  arXiv  Detail & Related papers  (2022-10-05T17:05:56Z)
• Neural Transition System for End-to-End Opinion Role Labeling [13.444895891262844]
  Unified opinion role labeling (ORL) aims to detect all possible opinion structures of opinion-holder-target' in one shot, given a text. We propose a novel solution by revisiting the transition architecture, and augment it with a pointer network (PointNet) The framework parses out all opinion structures in linear-time complexity, breaks through the limitation of any length of terms with PointNet.
  arXiv  Detail & Related papers  (2021-10-05T12:45:59Z)
• House-GAN++: Generative Adversarial Layout Refinement Networks [37.60108582423617]
  Our architecture is an integration of a graph-constrained GAN and a conditional GAN, where a previously generated layout becomes the next input constraint. A surprising discovery of our research is that a simple non-iterative training process, dubbed component-wise GT-conditioning, is effective in learning such a generator.
  arXiv  Detail & Related papers  (2021-03-03T18:15:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.