EcoScratch: Cost-Effective Multimodal Repair for Scratch Using Execution Feedback

• URL: http://arxiv.org/abs/2603.29624v1
• Date: Tue, 31 Mar 2026 11:45:36 GMT
• Title: EcoScratch: Cost-Effective Multimodal Repair for Scratch Using Execution Feedback
• Abstract summary: We present EcoScratch, a repair pipeline that uses lightweight runtime signals to decide whether the next attempt stays text-only or escalates to a multimodal prompting.<n>We evaluate 12 models on 100 executable Scratch repair projects under four controller settings, yielding 4800 repair trajectories.<n>It reaches the highest generation success (30.3%) while using less average cost and local-runtime energy than the two non-adaptive multimodal baselines under the same bounded trajectory budget; text-only lowest-cost floor.
• Score: 3.6908036186618314
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Scratch is the most popular programming environment for novices, with over 1.15 billion projects created worldwide. Unlike traditional languages, correctness in Scratch is defined by visible behavior on the stage rather than by code structure alone, so programs that appear correct in the workspace can still fail at runtime due to timing, event ordering, or cross-sprite interactions. Visual execution evidence such as gameplay videos can therefore be essential for diagnosis and repair. However, capturing and processing this evidence inside an automated repair loop introduces substantial overhead. Probing execution, recording stage behavior, rebuilding executable .sb3 projects, and verifying candidate fixes consume time, monetary cost, and resources across an entire repair trajectory rather than a single model call. We present EcoScratch, a repair pipeline that uses lightweight runtime signals to decide whether the next attempt stays text-only or escalates to multimodal prompting. The controller also sets the JSON Patch budget and verification effort, so evidence choice and repair budget are coupled inside the same decision. EcoScratch rebuilds candidate fixes into executable .sb3 projects and records per-trajectory traces, monetary cost, local-runtime energy. We evaluate 12 models on 100 executable Scratch repair projects under four controller settings, yielding 4800 repair trajectories. In this matrix, a selective multimodal policy gives the strongest observed success-cost-energy tradeoff. It reaches the highest generation success (30.3%) while using less average cost and local-runtime energy than the two non-adaptive multimodal baselines under the same bounded trajectory budget; text-only remains the lowest-cost floor. Across the evaluated matrix, multimodal evidence helps most when it is used to control escalation within a bounded trajectory budget rather than applied uniformly.

Related papers

• Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops [1.6093668627931699]
  Iterative repair loops have become a core design pattern in LLM-based software engineering systems.<n>We study repair-loop effectiveness across multiple software engineering tasks, including code generation, test generation, and code translation.<n>We observe a consistent pattern of diminishing returns: the first three to four repair iterations account for most achievable gains, while later iterations contribute only marginal improvements.
  arXiv  Detail & Related papers  (2026-07-06T15:14:10Z)
• A Stackelberg Framework for Resource-Aware LLM Agents: Learning, Repair, and Conditional Guarantees [1.9381445674403615]
  Large language model (LLM) agents increasingly operate as multi-turn systems must allocate context, prompt verbosity, and tool access under finite computational budgets.<n>We formulate resource governance as a contextual Stackelberg game: a controller commits to a quality target and a cost incentive, while an executor responds with resource actions over context, prompting, and tool usage.<n>We learn a conditional response model, optimize a leader policy against that model, and repair the resulting policy using real-API calibration and projection.
  arXiv  Detail & Related papers  (2026-06-22T08:38:22Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation [9.992155237142306]
  We introduce ClawTrace, an agent tracing platform that records every LLM call, tool use, and sub-agent spawn during an agent session.<n>CostCraft is a distillation pipeline that reads TraceCards and produces three types of skill patches.<n>We show that both cost attribution and prune patches independently reduce quality regressions.
  arXiv  Detail & Related papers  (2026-04-26T19:44:10Z)
• QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimization [50.722636027998]
  Large Language Models (LLMs) achieve strong program repair performance but often suffer from over-editing.<n>We propose PRepair, a framework that mitigates over-editing and improves repair accuracy.
  arXiv  Detail & Related papers  (2026-04-07T14:56:38Z)
• Runtime Execution Traces Guided Automated Program Repair with Multi-Agent Debate [8.424102114588559]
  Automated Program Repair (APR) struggles with complex logic errors and silent failures.<n>Current LLM-based APR methods are mostly static, relying on source code and basic test outputs.<n>We propose TraceRepair, a multi-agent framework that leverages runtime facts as shared constraints for patch validation.
  arXiv  Detail & Related papers  (2026-04-03T02:23:25Z)
• FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repair [45.42550492783294]
  Multimodal Automated Program Repair (MAPR) extends traditional program repair by requiring models to jointly reason over source code, textual issue descriptions, and visual artifacts such as GUI screenshots.<n>We propose FailureMem, a multimodal repair framework that integrates three key mechanisms: a hybrid workflow-agent architecture that balances structured localization with flexible reasoning, active perception tools that enable region-level visual grounding, and a Failure Memory Bank that converts past repair attempts into reusable guidance.
  arXiv  Detail & Related papers  (2026-03-18T15:24:22Z)
• ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming [3.935975887408409]
  Scratch programs exhibit deeply nested, non-linear structures, event-driven sprites, and tight coupling between code and multimedia assets.<n>We introduce ScratchEval, the first executable benchmark designed to evaluate LLM-based repair for Scratch programs.<n>The benchmark is constructed through a human-in-the-loop pipeline combining automated project mining with expert validation of trigger-outcome semantics.
  arXiv  Detail & Related papers  (2026-01-31T14:44:22Z)
• Outcome-Conditioned Reasoning Distillation for Resolving Software Issues [49.16055123488827]
  We present an Outcome-Conditioned Reasoning Distillation(O-CRD) framework that uses resolved in-repository issues with verified patches as supervision.<n>Starting from a historical fix, the method reconstructs a stage-wise repair trace backward from the verified outcome.<n>On SWE-Bench Lite, this approach increases Pass@1 by 10.4% with GPT-4o, 8.6% with DeepSeek-V3, and 10.3% with GPT-5.
  arXiv  Detail & Related papers  (2026-01-30T18:25:39Z)
• The 4/$δ$ Bound: Designing Predictable LLM-Verifier Systems for Formal Method Guarantee [5.345468714252351]
  This work bridges the gap by developing an LLM-Verifier Convergence Theorem.<n>We model the interaction between the LLM and the verifier as a discrete-time Markov Chain.<n>We stress-tested this prediction in an extensive empirical campaign comprising more than 90,000 trials.
  arXiv  Detail & Related papers  (2025-11-30T22:19:09Z)
• GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents [59.107657859025586]
  GUI-360$circ$ is a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs)<n>The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications.<n>The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space.
  arXiv  Detail & Related papers  (2025-11-06T12:19:02Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)
• Boosting Redundancy-based Automated Program Repair by Fine-grained Pattern Mining [18.7107522872479]
  We propose a new repair technique named Repatt, which incorporates a two-level pattern mining process for guiding effective patch generation.<n>We have conducted an experiment on the widely-used Defects4J benchmark and compared Repatt with ten state-of-the-art APR approaches.
  arXiv  Detail & Related papers  (2023-12-26T08:42:32Z)
• Fast Online Value-Maximizing Prediction Sets with Conformal Cost Control [63.90454433380153]
  Many real-world multi-label prediction problems involve set-valued predictions that must satisfy specific requirements dictated by downstream usage. We focus on a typical scenario where such requirements, separately encoding $textitvalue$ and $textitcost$, compete with each other. We present a general pipeline, dubbed as FavMac, to maximize the value while controlling the cost in such scenarios.
  arXiv  Detail & Related papers  (2023-02-02T02:53:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.