Mind the Gap: Diagnosing Constraint Discovery Failures in Text-in-Image Editing

• URL: http://arxiv.org/abs/2606.15982v1
• Date: Sun, 14 Jun 2026 19:14:25 GMT
• Title: Mind the Gap: Diagnosing Constraint Discovery Failures in Text-in-Image Editing
• Abstract summary: Key challenge in multimodal reasoning is determining which visual dependencies become relevant under a specific task.<n>We study this through edit-induced constraint discovery in text-in-image editing.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A key challenge in multimodal reasoning is determining which visual dependencies become relevant under a specific task, rather than merely recognizing visible content. We study this through edit-induced constraint discovery in text-in-image editing, a controlled diagnostic setting where a local text change can activate secondary consistency constraints: given a valid editing instruction and an image, can a model identify the secondary regions that must also change? Across 461 diagnostic cases, four MLLMs, and 19 constraint subtypes, models recover only 46% case-level macro recall under unguided prompting versus 94% when constraints are explicitly provided, suggesting that a substantial portion of the failure arises when models must decide which unstated dependencies to surface. Oracle-field decomposition shows that case-specific causal explanations are the most effective partial guidance (0.782 recall), above region names (0.610) or type labels (0.646), suggesting that edit-specific causal cues account for much of the oracle gain. A downstream experiment further shows that higher self-discovery recall does not necessarily improve task performance: unverified self-discovery introduces false positives that offset recall gains, motivating precision-aware constraint elicitation.

Related papers

• IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications [52.570108663867046]
  A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.<n>We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.<n>We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  arXiv  Detail & Related papers  (2026-09-09T17:59:04Z)
• When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory [0.0]
  Provenance links keep the evidence behind an inherited belief reachable; an agent with a verification budget must still choose which links to inspect.<n>We study a consolidated memory that states a decision constraint and whose source record has since been superseded by a record that withdraws it.<n>16 language models rarely re-verified a constraint that read as settled.<n>Once the constraint had been superseded, produced stale-consistent decisions in 77.3%, 74.7% and 74.7% of episodes across a primary run, a replication and a held-out domain.
  arXiv  Detail & Related papers  (2026-08-26T09:04:21Z)
• SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification [71.9783246097687]
  SEER hides candidate relations during pair localization, constructs a query-specific view with explicit subject/object roles, and retains the full image and sparse box geometry as complementary evidence.<n>For relation-choice protocols with exact inverse support, an optional refinement swaps the entity roles and changes the forward decision only when exactly one visual state obeys the corresponding inverse relation.<n>The unchanged protocol yields +4.35 to +11.79 on all 2,434 filtered EmbSpatial pair-relation questions across three models.
  arXiv  Detail & Related papers  (2026-08-04T13:16:15Z)
• Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets [13.85834524293015]
  We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with.<n>Models judge whether a candidate belongs far better than they author the set itself.
  arXiv  Detail & Related papers  (2026-08-02T05:00:44Z)
• Auditable Release Control for Pedagogical Leakage in LLM Tutors [0.0]
  Large language model tutors can be correct and helpful yet disclose an answer or decisive reasoning before that disclosure is authorized.<n>We formalize this state- and action-dependent failure as pedagogical leakage-aware authorization and introduce complete-mediation boundary.<n>A global A1 yields 0 majority and 54 any-utility flags, outperforming fitted Q on automatic safety and utility.
  arXiv  Detail & Related papers  (2026-08-01T08:24:30Z)
• Evidence-Grounded Constraint Checking in Construction Documents [1.3787753513429983]
  We present an evidence-grounded pipeline that normalizes extracted facts and executes four-state rules deterministically.<n>We evaluate its PDF evidence allocator on 160 reference-based tasks from 29 construction projects.
  arXiv  Detail & Related papers  (2026-07-31T06:25:41Z)
• DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules [4.124344125532972]
  Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger based on sensor conditions.<n> translating rules into maintenance steps requires asset-specific knowledge gained through years of practice.<n>We investigate whether LLMs can serve as decision support for this rule-to-action step and introduce ours, a benchmark of 6,690 expert-validated multiple-choice questions.
  arXiv  Detail & Related papers  (2026-05-09T02:17:39Z)
• Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation [0.0]
  We introduce DriftBench, a benchmark for evaluating constraint adherence in scientific ideation.<n>We find that iterative pressure reliably increases structural complexity and often reduces adherence to original constraints.<n>We release all briefs, prompts, rubrics, transcripts, and scores as an open benchmark.
  arXiv  Detail & Related papers  (2026-04-30T15:46:33Z)
• CCTU: A Benchmark for Tool Use under Complex Constraints [66.87622847854337]
  We introduce CCTU, a benchmark for evaluating large language models (LLMs) under complex constraints.<n>The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios.<n>We develop an executable constraint validation module that performs step-level validation and enforces compliance.
  arXiv  Detail & Related papers  (2026-03-16T14:05:13Z)
• Did You Forget What I Asked? Prospective Memory Failures in Large Language Models [0.0]
  Large language models often fail to satisfy formatting instructions when they must simultaneously perform demanding tasks.<n>We study this behaviour through a prospective memory inspired lens from cognitive psychology, using a controlled paradigm.<n>Vulnerability is highly type-dependent, with drops up to 50%, while avoidance constraints remain comparatively robust.<n>A salience-enhanced format (explicit instruction framing plus a trailing reminder) recovers much of the lost compliance, restoring performance to 90-100% in many settings.
  arXiv  Detail & Related papers  (2026-03-07T05:58:19Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models [48.361839372110246]
  We develop an automated instruction generation pipeline that performs constraint expansion, conflict detection, and instruction rewriting.<n>We evaluate 19 large language models and uncover substantial variation in performance across constraint forms.<n>In-depth analysis indicates that these gains stem primarily from modifications in the model's attention modules parameters.
  arXiv  Detail & Related papers  (2025-05-12T14:16:55Z)
• Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model [86.9619638550683]
  Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data.<n>However, these models display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of decision shortcuts''
  arXiv  Detail & Related papers  (2024-03-01T09:01:53Z)
• Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models [38.79074982172423]
  We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text. We propose modeling factual queries as constraint satisfaction problems. We find a strong positive relationship between the LLM's attention to constraint tokens and the factual accuracy of generations.
  arXiv  Detail & Related papers  (2023-09-26T17:48:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.