PaintBench: Deterministic Evaluation of Precise Visual Editing
- URL: http://arxiv.org/abs/2606.00188v1
- Date: Fri, 29 May 2026 16:01:14 GMT
- Title: PaintBench: Deterministic Evaluation of Precise Visual Editing
- Abstract summary: PaintBench is a benchmark for 20 fundamental precise visual editing operations.<n>Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1%.<n>Fine-grained benchmark diagnostics show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size.
- Score: 50.50559031686293
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores ($R^2 = 0.91$, $p < 0.001$). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.
Related papers
- FED-Bench: A Cross-Granular Benchmark for Disentangled Evaluation of Facial Expression Editing [29.7144418122336]
FED-Bench is a comprehensive benchmark featuring rigorous testing and an accurate evaluation suite.<n>We benchmark 18 image editing models, revealing that current approaches struggle to simultaneously achieve high fidelity and accurate expression manipulation.<n>Our benchmark and related code will be made publicly open soon.
arXiv Detail & Related papers (2026-03-31T12:52:17Z) - GEditBench v2: A Human-Aligned Benchmark for General Image Editing [58.86807672117726]
We introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks.<n>We also propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines.<n> PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average.
arXiv Detail & Related papers (2026-03-30T15:08:32Z) - InEdit-Bench: Benchmarking Intermediate Logical Pathways for Intelligent Image Editing Models [17.680767010203308]
We introduce InEdit-Bench, the first evaluation benchmark dedicated to reasoning over intermediate pathways in image editing.<n>InEdit-Bench comprises meticulously annotated test cases covering four fundamental task categories: state transition, dynamic process, temporal sequence, and scientific simulation.<n>Our comprehensive evaluation of 14 representative image editing models on InEdit-Bench reveals significant and widespread shortcomings in this domain.
arXiv Detail & Related papers (2026-03-04T02:24:43Z) - I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models [78.62380562116135]
Existing image editing benchmarks suffer from limited task scopes, insufficient evaluation dimensions, and heavy reliance on manual annotations.<n>We propose textbfI2I-Bench, a comprehensive benchmark for image-to-image editing models, which features 10 task categories across both single-image and multi-image editing tasks.<n>Using I2I-Bench, we benchmark numerous mainstream image editing models, investigating the gaps and trade-offs between editing models across various dimensions.
arXiv Detail & Related papers (2025-12-04T10:44:07Z) - Charts Are Not Images: On the Challenges of Scientific Chart Editing [66.38730113476677]
textitFigEdit is a benchmark for scientific figure editing comprising over 30,000 samples.<n>Our benchmark demonstrates the profound limitations of pixel-level manipulation.<n>By releasing textitFigEdit, we aim to enable systematic progress in structure-aware figure editing.
arXiv Detail & Related papers (2025-11-30T06:13:48Z) - Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing [84.16442052968615]
We introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE)<n>RISEBench focuses on four key reasoning categories: Temporal, Causal, Spatial, and Logical Reasoning.<n>We conduct experiments evaluating nine prominent visual editing models, comprising both open-source and proprietary models.
arXiv Detail & Related papers (2025-04-03T17:59:56Z) - DisPositioNet: Disentangled Pose and Identity in Semantic Image
Manipulation [83.51882381294357]
DisPositioNet is a model that learns a disentangled representation for each object for the task of image manipulation using scene graphs.
Our framework enables the disentanglement of the variational latent embeddings as well as the feature representation in the graph.
arXiv Detail & Related papers (2022-11-10T11:47:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.