CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

• URL: http://arxiv.org/abs/2606.00931v1
• Date: Sat, 30 May 2026 23:37:55 GMT
• Title: CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences
• Abstract summary: We introduce CV-Arena, an open benchmark designed to evaluate this capability at professional scales.<n>CV-Arena contains 12K high-resolution real-image instruction pairs spanning 16 instruction-based visual task types.<n>We develop CV-Agent, a lightweight agentic model that combines planning, editing, and verification.
• Score: 51.86576932800981
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Instruction-guided image editing is becoming a general interface for visual work, yet existing benchmarks still focus largely on narrow appearance edits and do not fully capture the diversity of real-image tasks in professional workflows. Here, we define instructional computer vision problem solving as a broader formulation of image editing: given a real input image and a natural-language instruction, a system must produce an edited output that realizes the requested transformation while satisfying explicit preservation, geometric, physical, and usability constraints. We introduce CV-Arena, an open benchmark designed to evaluate this capability at professional scales. CV-Arena contains 12K high-resolution real-image instruction pairs spanning 16 instruction-based visual task types, constructed using CogRetriever, a dual-track retrieval-and-curation pipeline that combines targeted web search, agentic query refinement, verification, and traceability. To evaluate models at scale while preserving human fidelity, we propose Active Elo, a human-AI collaborative preference protocol that leverages CV-Judge, a logic-gated, multi-dimensional VLM evaluator, to reject clear failures and resolve high-confidence comparisons; and to route close, high-quality comparisons to expert raters. Mixed human and AI supervision is then aggregated through reliability-weighted Elo updates. Our comprehensive evaluation of 21 systems, including proprietary, open-source, and agentic models, on CV-Arena reveals persistent gaps in instruction adherence, physical reasoning, structural control, and fine-grained detail preservation. We further develop CV-Agent, a lightweight agentic model that combines planning, editing, and verification, and demonstrate that closed-loop reasoning is a promising direction for professional-grade instruction-following visual editing.

Related papers

• DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable [10.070492917829995]
  We introduce DrawAI, comprising an agentic benchmark, DrawAI-Bench, and a reconstruction workflow, DrawAI-Flow.<n>We formulate image-to-editable reconstruction, which recovers a structured, directly manipulable artifact from an image while preserving its visual and semantic content.<n>The results show that reconstruction quality and costs vary substantially across model-harness configurations, while DrawAI-Flow consistently improves editable structure.
  arXiv  Detail & Related papers  (2026-08-01T09:05:06Z)
• GMO-E$^2$DIT: Grounded Multi-Operation Editing for E-Commerce Images [16.213122519512932]
  Real-world e-commerce image editing often requires multiple, localized, and auditable operations rather than global restyling.<n>Existing one-shot editors conflate intent resolution, spatial grounding, and synthesis into a single step.<n>We introduce GMO-E$2$DIT, an agentic editing framework that couples a Vision-Language Model (VLM) with a mask-conditioned image editor to tackle structured multi-turn task completion.
  arXiv  Detail & Related papers  (2026-07-01T13:23:45Z)
• Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking [14.173267480504842]
  Scribble-guided image editing allows users to combine simple scribble annotations with text prompts to specify both where and how an image should be edited.<n>Existing models still exhibit unstable performance under this paradigm, especially in multi-task scenarios.<n>We conduct empirical studies using an open-source editing model and reveal an asymmetry in generalization.
  arXiv  Detail & Related papers  (2026-05-25T08:20:23Z)
• CAMEO: A Conditional and Quality-Aware Multi-Agent Image Editing Orchestrator [8.901958956012072]
  Conditional image editing is crucial in scenarios requiring strict structural control.<n>Most approaches rely on single-step generation.<n>We propose textbfCAMEO, a structured multi-agent framework that reformulates conditional editing as a quality-aware, feedback-driven process.
  arXiv  Detail & Related papers  (2026-04-03T16:27:02Z)
• How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing [56.60465182650588]
  We introduce three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning.<n>We propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment.<n>We find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models.
  arXiv  Detail & Related papers  (2026-02-02T09:24:45Z)
• Image-POSER: Reflective RL for Multi-Expert Image Generation and Editing [16.943575863059607]
  Image-POSER orchestrates a diverse registry of pretrained text-to-image and image-to-image experts.<n>It handles long-form prompts end-to-end through dynamic task decomposition.<n>It is consistently preferred in human evaluations.
  arXiv  Detail & Related papers  (2025-11-15T03:15:34Z)
• EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing [170.71134330650796]
  EdiVal-Agent is an object-centric evaluation framework for instruction-based image editing.<n>It is designed to assess not only standard single-turn but also multi-turn instruction-based editing with precision.<n>We build EdiVal-Bench, a benchmark covering 9 instruction types and 13 state-of-the-art editing models spanning in-context, flow-matching, and diffusion paradigms.
  arXiv  Detail & Related papers  (2025-09-16T17:45:39Z)
• What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models [88.398085358514]
  DICE is a model designed to detect localized differences between the original and the edited image.<n>It is trained using a strategy that leverages self-supervision, distillation from inpainting networks, and full supervision.<n>We demonstrate that DICE effectively identifies coherent edits, effectively evaluating images generated by different editing models with a strong correlation with human judgment.
  arXiv  Detail & Related papers  (2025-05-26T18:00:10Z)
• Ranking-aware adapter for text-driven image ordering with CLIP [76.80965830448781]
  We propose an effective yet efficient approach that reframes the CLIP model into a learning-to-rank task.<n>Our approach incorporates learnable prompts to adapt to new instructions for ranking purposes.<n>Our ranking-aware adapter consistently outperforms fine-tuned CLIPs on various tasks.
  arXiv  Detail & Related papers  (2024-12-09T18:51:05Z)
• PromptFix: You Prompt and We Fix the Photo [84.69812824355269]
  Diffusion models equipped with language models demonstrate excellent controllability in image generation tasks. The lack of diverse instruction-following data hampers the development of models. We propose PromptFix, a framework that enables diffusion models to follow human instructions.
  arXiv  Detail & Related papers  (2024-05-27T03:13:28Z)
• SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning [69.93418696660785]
  Iterative Language-Based Image Editing (IL-BIE) tasks follow iterative instructions to edit images step by step.<n>Data scarcity is a significant issue for ILBIE as it is challenging to collect large-scale examples of images before and after instruction-based changes.<n>We introduce a Self-Supervised Counterfactual Reasoning framework that incorporates counterfactual thinking to overcome data scarcity.
  arXiv  Detail & Related papers  (2020-09-21T01:45:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.