Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2

• URL: http://arxiv.org/abs/2606.31543v1
• Date: Tue, 30 Jun 2026 11:55:33 GMT
• Title: Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2
• Abstract summary: I present a solver for ARC-AGI-2, a few-shot visual reasoning benchmark.<n>It generates diverse candidates independently across text, image, and code channels.<n>Unlike self-consistency or majority voting, it reliably recovers correct minority hypotheses on tasks where the modal answer is wrong.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models can produce fluent, internally coherent reasoning traces for abstract reasoning tasks while still being confidently wrong - making selection among candidates, not just generation, the central challenge. I present a solver for ARC-AGI-2, a few-shot visual reasoning benchmark, built around two principles: (i) treating reasoning modalities as search operators, generating diverse candidates independently across text, image, and code channels, and (ii) context-preserving holistic judging, in which a judge model jointly compares all candidate reasoning traces within a single long-context prompt. Unlike self-consistency or majority voting, this approach reliably recovers correct minority hypotheses on tasks where the modal answer is wrong. On the ARC Prize semi-private evaluation set, the solver achieves 72.9 percent at USD 38.99 per task - the highest score on the verified leaderboard at the time of writing, exceeding the best standalone frontier models, GPT-5.2 Pro at 54.2 percent and Gemini 3 Pro at 54.0 percent, by +18.7 percentage points. On the public evaluation set, it achieves 76.1 percent at USD 19.69 per task. I release the full source code and document extensive negative results, including the finding that prescriptive prompting templates and iterative refinement systematically reduce hypothesis diversity and degrade performance.

Related papers

• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback [4.062253229926774]
  A-ProS is an autonomous AI agent that solves competitive programming problems through a hybrid multi-model feedback framework.<n>A-ProS achieves over 2x greater gains compared to baseline agent loops.
  arXiv  Detail & Related papers  (2026-05-18T08:55:30Z)
• Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty? [59.923111838399144]
  We introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter.<n>VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task.<n>We find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts.
  arXiv  Detail & Related papers  (2026-05-12T19:33:28Z)
• PinPoint: Evaluation of Composed Image Retrieval with Explicit Negatives, Multi-Image Queries, and Paraphrase Testing [3.889218009169166]
  Composed Image Retrieval (CIR) has made significant progress, yet current benchmarks are limited to single ground-truth answers.<n>We present PinPoint, a comprehensive real world benchmark with 7,635 queries and 329K relevance judgments.
  arXiv  Detail & Related papers  (2026-03-04T20:55:30Z)
• Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning [0.0]
  We introduce Pencil Puzzle Bench, a framework for evaluating large language model reasoning through pencil puzzles.<n>From a database of 62,231 puzzles, we select a benchmark of 300 puzzles spanning 20 varieties and evaluate 51 models from 11 providers.<n>A key differentiator of our benchmark is that every intermediate board state can be checked against variety-specific constraints.
  arXiv  Detail & Related papers  (2026-03-02T17:40:54Z)
• EdgeJury: Cross-Reviewed Small-Model Ensembles for Truthful Question Answering on Serverless Edge Inference [0.0]
  We present EdgeJury, a lightweight ensemble framework that improves truthfulness and robustness.<n>On TruthfulQA (MC1), EdgeJury achieves 76.2% accuracy.<n>On a 200-question adversarial EdgeCases set, EdgeJury yields +48.2% relative gains.
  arXiv  Detail & Related papers  (2025-12-29T14:48:40Z)
• LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads? [5.835205320809048]
  LiveOIBench is a benchmark featuring 403 Olympiad-level competitive programming problems with an average of 60 expert-designed test cases.<n>The problems are sourced directly from 72 official Informatics Olympiads in different regions conducted between 2023 and 2025.<n>LiveOIBench distinguishes itself through four key features: meticulously curated high-quality tasks with detailed subtasks and extensive private test cases.
  arXiv  Detail & Related papers  (2025-10-10T17:54:24Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D [97.08549913899247]
  We introduce Contra4, a dataset for contrastive cross-modal reasoning across four modalities: image, audio, video, and 3D.<n>Each example presents a natural language question alongside multiple candidate modality instances, and the model must select the one that semantically aligns with the prompt.<n>Contra4 combines human-annotated captions with a mixture-of-models round-trip-consistency filter to ensure high-quality supervision, resulting in 174k training examples and a manually verified test set of 2.3k samples.
  arXiv  Detail & Related papers  (2025-06-02T03:12:13Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.