Learning CLI Agents with Structured Action Credit under Selective Observation

• URL: http://arxiv.org/abs/2605.08013v1
• Date: Fri, 08 May 2026 17:02:31 GMT
• Title: Learning CLI Agents with Structured Action Credit under Selective Observation
• Abstract summary: Command line interface (CLI) agents are emerging as a practical paradigm for agent-computer interaction over evolvings, executable command line programs, and online execution feedback.<n>Recent work has used reinforcement learning (RL) to learn these interaction abilities from verifiable task feedback, yet few methods exploit the native structured attributes of CLI actions as learning signals.<n>We study these bottlenecks through shell-driven information extraction and file editing tasks.
• Score: 10.420078730796321
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Command line interface (CLI) agents are emerging as a practical paradigm for agent-computer interaction over evolving filesystems, executable command line programs, and online execution feedback. Recent work has used reinforcement learning (RL) to learn these interaction abilities from verifiable task feedback, yet few methods exploit the native structured attributes of CLI actions as learning signals. Beyond this underused action structure, CLI learning also couples two bottlenecks for coding agents. First, the agent must identify task-relevant evidence in a large codebase from partial observations. Second, sparse terminal rewards must be assigned to the actions that shape a long multi-turn trajectory. We study these bottlenecks through shell-driven information extraction and file editing tasks. For selective observation, we introduce $σ$-Reveal, an inference-time mechanism that selects token-budgeted context for the same CLI. For credit assignment, we propose Action Advantage Assignment ($\mathrm{A}^3$), a native agentic RL method that preserves the algorithmic complexity of standard agentic RL. $\mathrm{A}^3$ constructs turn-level advantages from episode-level relative feedback, abstract syntax tree (AST) based action sub-chain residuals, and tree-level trajectory margins. To further evaluate this problem setting, we construct ShellOps, a verifiable dataset suite covering CLI tasks in repository environments.

Related papers

• AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories [36.7302669545557]
  Real-world applications provide no pre-defined tasks or verifiers, no faithful simulators, and limited budget for large-scale environment interaction.<n>We propose textbfAgentBrew, an offline training framework that learns effective tool-use policies from a single batch of raw interaction trajectories.<n>AgentBrew improves Qwen3-32B by +8.7 Acc / +9.7 Score on average, surpassing Qwen3-235B (+2.3 / +4.4) and outperforming rejection sampling.
  arXiv  Detail & Related papers  (2026-09-05T03:04:23Z)
• PRIMA: Operational Patterns for Resilient Multi-Agent Research with Verifiable Identity and Convergent Feedback [0.0]
  PRIMA operates as coordinated multi-agent research systems over multi-hour runs.<n>Main contributions are three operational patterns for surviving failure modes.<n>A Graph Isomorphism case study grounds the architectural claims in a generated artifact.
  arXiv  Detail & Related papers  (2026-05-23T23:27:46Z)
• When LLMs Team Up: A Coordinated Attack Framework for Automated Cyber Intrusions [27.62993988829038]
  This paper presents CAESAR, a coordinated multi-agent framework for analysis of intrusion-style tasks.<n>CAESAR decomposes the workflow into five typed roles and coordinates them through a bounded protocol.<n>Compared with a single-agent baseline under matched budgets and tool access, CAESAR improves task success and reduces performance variance.
  arXiv  Detail & Related papers  (2026-05-09T07:47:42Z)
• ARISE: A Repository-level Graph Representation and Toolset for Agentic Fault Localization and Program Repair [0.0]
  We present ARISE (Agentic Repository-level Issue Solving Engine), which augments an LLM-based agent with a multi-granularity program graph.<n>ARISE exposes this graph through a three-tier tool API, which brings data-flow slicing as a first-class, queryable agent primitive.<n>We evaluate on SWE-bench Lite (300 real GitHub issues, 11 Python repositories) using Qwen2.5-Coder-32B-Instruct as the backbone.
  arXiv  Detail & Related papers  (2026-05-04T19:59:23Z)
• Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features [1.5874067490843806]
  Control Reinforcement Learning trains a policy to select SAE features for steering at each token, producing interpretable intervention logs.<n> Adaptive Feature Masking encourages diverse feature discovery while preserving singlefeature interpretability.<n>On Gemma 2 2B across MMLU, BBQ, GSM8K, HarmBench, and XSTest, CRL achieves improvements while providing per-token intervention logs.
  arXiv  Detail & Related papers  (2026-02-11T02:28:49Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAG [71.06073770344732]
  Agentic retrieval-augmented generation (RAG) formulates question answering as a multi-step interaction between reasoning and information retrieval.<n>We present TreePS-RAG, an online, tree-based RL framework for agentic RAG that enables step-wise credit assignment while retaining outcome-only rewards.
  arXiv  Detail & Related papers  (2026-01-11T14:07:30Z)
• Zero-Shot Instruction Following in RL via Structured LTL Representations [54.08661695738909]
  Linear temporal logic (LTL) is a compelling framework for specifying complex, structured tasks for reinforcement learning (RL) agents.<n>Recent work has shown that interpreting instructions as finite automata, which can be seen as high-level programs monitoring task progress, enables learning a single generalist policy capable of executing arbitrary instructions at test time.<n>We propose a novel approach to learning a multi-task policy for following arbitrary instructions that addresses this shortcoming.
  arXiv  Detail & Related papers  (2025-12-02T10:44:51Z)
• A Tale of LLMs and Induced Small Proxies: Scalable Agents for Knowledge Mining [52.57170634957137]
  Falconer is a framework that combines large language models with lightweight proxy models for scalable knowledge mining.<n>LLMs act as planners, decomposing user instructions into executable pipelines, and as annotators, generating supervision to train small proxies.<n>Experiments show that Falconer closely matches state-of-the-art LLMs in instruction-following accuracy while reducing inference cost by up to 90%.
  arXiv  Detail & Related papers  (2025-10-01T20:06:48Z)
• VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use [78.29315418819074]
  We introduce VerlTool, a unified and modular framework that addresses limitations through systematic design principles.<n>Our framework formalizes ARLT as multi-turn trajectories with multi-modal observation tokens (text/image/video), extending beyond single-turn RLVR paradigms.<n>The modular plugin architecture enables rapid tool integration requiring only lightweight Python definitions.
  arXiv  Detail & Related papers  (2025-09-01T01:45:18Z)
• Language Models can Solve Computer Tasks [13.914130729517584]
  We show that a pre-trained large language model (LLM) agent can execute computer tasks guided by natural language using a simple prompting scheme. We compare multiple LLMs and find that RCI with the InstructGPT-3+RLHF LLM is state-of-the-art on MiniWoB++.
  arXiv  Detail & Related papers  (2023-03-30T16:01:52Z)
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning [102.36450942613091]
  We propose an inverse reinforcement learning algorithm, called emphinverse temporal difference learning (ITD) We show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called $Psi Phi$-learning.
  arXiv  Detail & Related papers  (2021-02-24T21:12:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.