Discovering Reinforcement Learning Interfaces with Large Language Models

• URL: http://arxiv.org/abs/2605.03408v1
• Date: Tue, 05 May 2026 06:27:20 GMT
• Title: Discovering Reinforcement Learning Interfaces with Large Language Models
• Abstract summary: Reinforcement learning systems rely on environment interfaces that specify observations and reward functions.<n>Recent work has automated reward design using large language models (LLMs), but these approaches assume fixed observations.<n>We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated.
• Score: 0.9558392439655014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning systems rely on environment interfaces that specify observations and reward functions, yet constructing these interfaces for new tasks often requires substantial manual effort. While recent work has automated reward design using large language models (LLMs), these approaches assume fixed observations and do not address the broader challenge of synthesizing complete task interfaces. We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated. We propose LIMEN (Code available at https://github.com/Lossfunk/LIMEN), a LLM guided evolutionary framework that produces candidate interfaces as executable programs and iteratively refines them using policy training feedback. Across novel discrete gridworld tasks and continuous control domains spanning locomotion and manipulation, joint evolution of observations and rewards discovers effective interfaces given only a trajectory-level success metric, while optimizing either component alone fails on at least one domain. These results demonstrate that automatic construction of RL interfaces from raw state can substantially reduce manual engineering and that observation and reward components often benefit from co-design, as single-component optimization fails catastrophically on at least one domain in our evaluation suite.

Related papers

• Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation [23.371552518874807]
  Automated Reward Design (ARD) aims to replace manual reward engineering in reinforcement learning with language-driven reward function synthesis.<n>Existing approaches based on large language models (LLMs) remain inherently correlation-driven, relying on iterative environmental feedback to refine reward hypotheses for each specific task.<n>We propose the Causal Reward World Model (CRWM), which explicitly models the causal relationships between candidate reward components and task-targeted physical variables through offline pre-training on multi-task interaction data.
  arXiv  Detail & Related papers  (2026-06-22T12:57:17Z)
• LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning [11.35341244051162]
  Current on-device vision-language GUI agents are constrained by limited model capacity.<n>TraditionalSupervised Fine-Tuning for small-scale models often leads to overfitting, catastrophic forgetting and policy rigidity.<n>We propose a novel SFT-free training paradigm that significantly enhances the performance of small-scale models.
  arXiv  Detail & Related papers  (2026-05-08T09:38:29Z)
• Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards [76.49428173793386]
  LLMs must invoke multiple dependent APIs in the correct order while propagating intermediate outputs.<n>Training models to handle such faces two obstacles: existing environments focus on simple per-turn function calls with simulated data, and binary rewards.<n>We present a framework addressing both challenges. First, we construct a reinforcement learning environment backed by a large-scale cache of real API responses, enabling a data synthesis pipeline that samples valid multi-step orchestration traces.<n>Second, we propose a graduated reward design that decomposes correctness into atomic validity and orchestration.
  arXiv  Detail & Related papers  (2026-03-25T18:31:39Z)
• What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces? [32.72487312750579]
  We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces.<n>Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting.<n>We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces.
  arXiv  Detail & Related papers  (2026-02-02T04:02:03Z)
• AR-MOT: Autoregressive Multi-object Tracking [56.09738000988466]
  We propose a novel autoregressive paradigm that formulates MOT as a sequence generation task within a large language model (LLM) framework.<n>This design enables the model to output structured results through flexible sequence construction, without requiring any task-specific heads.<n>To enhance region-level visual perception, we introduce an Object Tokenizer based on a pretrained detector.
  arXiv  Detail & Related papers  (2026-01-05T09:17:28Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)
• ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search [53.40810298627443]
  ReGUIDE is a framework for web grounding that enables MLLMs to learn data efficiently through self-generated reasoning and spatial-aware criticism.<n>Our experiments demonstrate that ReGUIDE significantly advances web grounding performance across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-05-21T08:36:18Z)
• Towards Unified Token Learning for Vision-Language Tracking [65.96561538356315]
  We present a vision-language (VL) tracking pipeline, termed textbfMMTrack, which casts VL tracking as a token generation task. Our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target.
  arXiv  Detail & Related papers  (2023-08-27T13:17:34Z)
• Soft Hierarchical Graph Recurrent Networks for Many-Agent Partially Observable Environments [9.067091068256747]
  We propose a novel network structure called hierarchical graph recurrent network(HGRN) for multi-agent cooperation under partial observability. Based on the above technologies, we proposed a value-based MADRL algorithm called Soft-HGRN and its actor-critic variant named SAC-HRGN.
  arXiv  Detail & Related papers  (2021-09-05T09:51:25Z)
• A Unified Object Motion and Affinity Model for Online Multi-Object Tracking [127.5229859255719]
  We propose a novel MOT framework that unifies object motion and affinity model into a single network, named UMA. UMA integrates single object tracking and metric learning into a unified triplet network by means of multi-task learning. We equip our model with a task-specific attention module, which is used to boost task-aware feature learning.
  arXiv  Detail & Related papers  (2020-03-25T09:36:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.