In harmony with gpt-oss

• URL: http://arxiv.org/abs/2604.00362v1
• Date: Wed, 01 Apr 2026 01:16:13 GMT
• Title: In harmony with gpt-oss
• Abstract summary: No one has independently reproduced OpenAI's published scores for GPt-oss-20b with tools.<n>We reverse-engineered the model's in-distribution tools.<n>We then built a native harmony agent harness.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: No one has independently reproduced OpenAI's published scores for gpt-oss-20b with tools, because the original paper discloses neither the tools nor the agent harness. We reverse-engineered the model's in-distribution tools: when prompted without tool definitions, gpt-oss still calls tools from its training distribution with high statistical confidence -- a strong prior, not a hallucination. We then built a native harmony agent harness (https://github.com/borislavmavrin/harmonyagent.git) that encodes messages in the model's native format, bypassing the lossy Chat Completions conversion. Together, these yield the first independent reproduction of OpenAI's published scores: 60.4% on SWE Verified HIGH (published 60.7%), 53.3% MEDIUM (53.2%), and 91.7% on AIME25 with tools (90.4%).

Related papers

• NovaFabric: Tamper-Evident, Replayable Evidence for Autonomous AI Agent Runs [0.0]
  We present NovaFabric, producing audit-grade execution evidence.<n>It records an agent run, without modifying agent logic, into a portable Run Capsule.<n>Sealed runs are re-executable under a four-mode replay protocol.<n>We evaluate eight research questions at measured scope.
  arXiv  Detail & Related papers  (2026-09-11T08:35:43Z)
• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices [48.36933831982682]
  We introduce ADeptS-Bench, a trustworthiness benchmark grounded in the ADEPTS capability framework.<n>The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface.<n>The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous.
  arXiv  Detail & Related papers  (2026-08-25T23:21:57Z)
• Joint Optimization of Tool Creation and Use for Large Language Model Agents [69.07066297088187]
  SMITH (-grounded Multi-task Iterative Tool Honing) trains tool creation and tool use inside a single policy.<n>A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks.<n>It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline)
  arXiv  Detail & Related papers  (2026-08-25T13:59:34Z)
• Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM [1.3660182208844809]
  This paper reports on the AI-assisted construction of PM4Py-UCM, an open-source tool that mines Use Case Map (UCM) models from event logs.<n>We characterize the development record itself, composed of 18 agent sessions (374 human turns and 10,328 tool actions over 65 hours), 151 commits, 20 releases, and a test suite grown from 108 to 691 test functions.<n>Up to version 0.7.4, fixes outnumber features 2.3:1, with 18% of turns for correcting agent errors.
  arXiv  Detail & Related papers  (2026-07-30T20:27:32Z)
• When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More [10.283803346813047]
  A growing line of work equips large language model (LLM) agents with graph neural networks (GNNs) as callable tools.<n>We expose a frozen GNN to a ReAct-style LLM agent as an explicit tool and measure, on node classification over a text-attributed graph.<n>We find the agent does not exercise judgment: its predictions agree with the raw GNN's 97.6-99.2% of the time (5 seeds)<n>We find the cost of deference does not shrink as capability grows and grows where alternatives emerge.
  arXiv  Detail & Related papers  (2026-06-12T14:13:19Z)
• When to Trust Tools? Adaptive Tool Trust Calibration For Tool-Integrated Math Reasoning [58.75883713573783]
  Tool-Integrated Reasoning has emerged as a promising paradigm that incorporates tool call and execution within the reasoning trajectory.<n>We find that when the reasoning of the model conflicts with the tool results, the model tends to believe in its own reasoning.<n>We introduce Adaptive Tool Trust (ATTC), a framework that guides the model to adaptively choose to trust or ignore the tool results.
  arXiv  Detail & Related papers  (2026-04-09T14:14:37Z)
• Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation [0.0]
  Pashto is spoken by approximately 60--80 million people but has no published benchmarks for multilingual automatic speech recognition (ASR) on any shared public test set.<n>This paper reports the first reproducible multi-model evaluation on public Pashto data, covering zero-shot ASR, script-level failure, and cross-domain evaluation of fine-tuned models.
  arXiv  Detail & Related papers  (2026-04-06T11:23:42Z)
• Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents [0.0]
  NabaOS is a lightweight verification framework for interactive agents.<n>It classifies every claim in an AI response by its source.<n>It detects 94.2% of fabricated tool executions, 87.6% of misstating counts, and 78.4% of false absence claims.
  arXiv  Detail & Related papers  (2026-03-09T20:45:41Z)
• Pre-AI Baseline: Developer IDE Satisfaction and Tool Autonomy in 2022 [0.0]
  This study analyzes satisfaction data from 1,155 software developers collected in July 2022, immediately preceding the mainstream adoption of generative AI tools.<n>We report a high-satisfaction ecosystem, dominated by Visual Studio Code (79% usage)<n>We identify an "experimenter" segment (29.9%) characterized by high tool churn but no significant satisfaction difference.
  arXiv  Detail & Related papers  (2026-03-06T09:01:53Z)
• AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoning [36.67330306977483]
  Large Audio Language Models (LALMs) excel at perception but struggle with complex reasoning requiring precise acoustic measurements.<n>We propose AuTAgent, a reinforcement learning framework that learns when and which tools to invoke.
  arXiv  Detail & Related papers  (2026-02-14T09:12:20Z)
• Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing [16.839489120513505]
  InfTool orchestrates three collaborative agents to generate diverse, verified trajectories spanning single-turn calls to complex multi-step gated calls.<n>We show that InfTool transforms a base 32B model from 19.8% to 70.9% accuracy (+258%), surpassing models 10x larger and rivaling Claude-Opus.
  arXiv  Detail & Related papers  (2025-12-29T17:12:39Z)
• AlphaApollo: Orchestrating Foundation Models and Professional Tools into a Self-Evolving System for Deep Agentic Reasoning [110.57865233597762]
  We present AlphaApollo, a self-evolving agentic reasoning system.<n>It aims to address two bottlenecks in foundation model (FM) reasoning-limited model-intrinsic capacity and unreliable test-time computation.<n>AlphaApollo orchestrates multiple models with professional tools to enable deliberate, verifiable reasoning.
  arXiv  Detail & Related papers  (2025-10-05T15:42:24Z)
• MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools [54.63478102768333]
  Well-calibrated model confidences can be used to weigh the risk versus reward of potential actions.<n>We propose a novel class of model-internal confidence estimators (MICE) to better assess confidence when calling tools.
  arXiv  Detail & Related papers  (2025-04-28T18:06:38Z)
• SinaTools: Open Source Toolkit for Arabic Natural Language Processing [0.5461938536945721]
  We introduce SinaTools, an open-source Python package for Arabic natural language processing and understanding. This paper presents SinaTools and its benchmarking results, demonstrating that SinaTools outperforms all similar tools on the aforementioned tasks.
  arXiv  Detail & Related papers  (2024-11-03T11:03:52Z)
• Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play [52.3079697845254]
  eva is the first method that allows language models to adaptively create training prompts in both offline and online RL post-training.<n>We show eva can create effective RL curricula and is robust across ablations.
  arXiv  Detail & Related papers  (2024-10-31T08:15:32Z)
• Tool-Augmented Reward Modeling [58.381678612409]
  We propose a tool-augmented preference modeling approach, named Themis, to address limitations by empowering RMs with access to external environments. Our study delves into the integration of external tools into RMs, enabling them to interact with diverse external sources. In human evaluations, RLHF trained with Themis attains an average win rate of 32% when compared to baselines.
  arXiv  Detail & Related papers  (2023-10-02T09:47:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.