Sandboxed Coding Agents are Competitive Omni-modal Task Solvers

• URL: http://arxiv.org/abs/2606.00579v1
• Date: Sat, 30 May 2026 07:04:27 GMT
• Title: Sandboxed Coding Agents are Competitive Omni-modal Task Solvers
• Abstract summary: We show that coding agents with only text+image access and a sandboxed tool-use interface can match SOTA native omnimodal models.<n>Our trajectory analysis suggests that their strength comes from writing code and orchestrating tools to extract relevant evidence.<n>We introduce Code-X, a training recipe with the OmniCoding trajectory dataset and verifiable reward, and provide baselines on Qwen-3.5-9B and Qwen-3.6-27B.
• Score: 22.269016330825128
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native omnimodal models. We show that this is not always the case: coding agents with only text+image access and a sandboxed tool-use interface can match, and in several settings outperform, SOTA native omnimodal models and predefined multimodal agent scaffolds across multiple audio-video benchmarks. Our trajectory analysis suggests that their strength comes from writing code and orchestrating tools to extract relevant evidence from transcripts, frames, and other modality signals, thereby converting omnimodal tasks into retrieval and information-processing problems rather than ingesting entire media streams. We further characterize their limitations through a failure taxonomy and process-level trace analysis, and show that simple skill injection, including human-written and self-distilled skills, substantially improves performance. To explore open-source elicitation, we introduce Code-X, a training recipe with the OmniCoding trajectory dataset and verifiable reward, and provide baselines on Qwen-3.5-9B and Qwen-3.6-27B. Finally, we argue that the next frontier is many-modality processing, and introduce TerminalBench-O, a process-level benchmark for real-world omnimodal processing tasks. Code will be available at https://github.com/Dongping-Chen/OmniCoding.

Related papers

• Towards Long-horizon Agentic Multimodal Search [109.0092257657625]
  We propose a novel Long-horizon MultiModal deep search framework, named LMM-Searcher, centered on a file-based visual representation mechanism.<n>We equip the agent with a tailored fetch-image tool, enabling a progressive, on-demand visual loading strategy for active perception.<n>Our method successfully scales to 100-turn search horizons, achieving state-of-the-art performance among open-source models.
  arXiv  Detail & Related papers  (2026-04-14T15:40:28Z)
• OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments [5.613789567241434]
  We introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments.<n>The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications.<n>Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals.
  arXiv  Detail & Related papers  (2026-04-03T08:57:06Z)
• OmniTrace: A Unified Framework for Generation-Time Attribution in Omni-Modal LLMs [31.589945976149973]
  We introduce OmniTrace, a lightweight and model-agnostic framework that formalizes attribution as a generation-time tracing problem.<n>We show that generation-aware span-level attribution produces more stable and interpretable explanations than naive self-attribution.<n>Our results suggest that treating attribution as a structured generation-time tracing problem provides a scalable foundation for transparency in omni-modal language models.
  arXiv  Detail & Related papers  (2026-03-20T17:25:00Z)
• A Versatile Multimodal Agent for Multimedia Content Generation [66.86040734610073]
  We propose a MultiMedia-Agent designed to automate complex content creation tasks.<n>Our agent system includes a data generation pipeline, a tool library for content creation, and a set of metrics for evaluating preference alignment.
  arXiv  Detail & Related papers  (2026-01-06T18:49:47Z)
• A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos [76.98722001848493]
  LongShOTBench is a diagnostic benchmark for long-form multimodal video understanding.<n>It includes open-ended, intent-driven questions; single- and multi-turn dialogues; and tasks requiring multimodal reasoning and agentic tool use.<n>LongShOTAgent is an agentic system that analyzes long videos via preprocessing, search, and iterative refinement.
  arXiv  Detail & Related papers  (2025-12-18T18:59:27Z)
• M3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark [45.755057449698825]
  M3-Bench is the first benchmark for evaluating multimodal tool use under the Model Context Protocol.<n>We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching.<n>The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification.
  arXiv  Detail & Related papers  (2025-11-21T19:27:02Z)
• What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities [56.646832992178105]
  We introduce OmniBench, a cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity.<n>We present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities.<n>Our dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate.
  arXiv  Detail & Related papers  (2025-06-10T15:59:38Z)
• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks [94.19506319646376]
  We introduce Agent-X, a benchmark for evaluating vision-centric agents in real-world, multimodal settings.<n>Agent-X features 828 agentic tasks with authentic visual contexts, including images, multi-image comparisons, videos, and instructional text.<n>Our results reveal that even the best-performing models, including GPT, Gemini, and Qwen families, struggle to solve multi-step vision tasks.
  arXiv  Detail & Related papers  (2025-05-30T17:59:53Z)
• OmniBench: Towards The Future of Universal Omni-Language Models [63.16606414452612]
  We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.<n>Our evaluation reveals that open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts.<n>We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance.
  arXiv  Detail & Related papers  (2024-09-23T17:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.