STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models

• URL: http://arxiv.org/abs/2607.18580v1
• Date: Mon, 20 Jul 2026 23:26:32 GMT
• Title: STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Models
• Abstract summary: Vision-language-action (VLA) models have shown impressive generalization, but often lack interpretability.<n>We propose a hierarchical framework that uses Signal Temporal Logic (STL) as a shared representation connecting high-level language understanding with low-level robot execution.<n>We evaluate the approach on a real-world tabletop domain, demonstrating how formal specifications can improve the precision, reliability, and interpretability of language-conditioned robot planning.
• Score: 9.560201418658144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language-action (VLA) models have shown impressive generalization, but often lack interpretability and can struggle to follow precise natural language instructions that encode spatial, temporal, and logical requirements. We propose a hierarchical framework that uses Signal Temporal Logic (STL) as a shared representation connecting high-level language understanding with low-level robot execution. A high-level policy leverages a VLM to decompose language instructions into high-level subtasks, generate STL specifications for each subtask, and choose a low-level policy for executing each subtask. The STL specifications translate language-derived intent into precise constraints, and the low-level policy selection determines whether those constraints are enforced directly through STL-guided model-predictive control or monitored during execution of a learned policy for perceptually complex, or contact-rich behaviors. By integrating STL into plan validation, low-level policy, subtask monitoring, and replanning, our framework enables language-derived plans to be checked, optimized, and revised at runtime using a common formal structure. We evaluate the approach on a real-world tabletop domain, demonstrating how formal specifications can improve the precision, reliability, and interpretability of language-conditioned robot planning.

Related papers

• Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking [61.91472604294714]
  We propose a novel language dependency parsing mechanism to distill tracking principal components.<n>We exploit the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL.<n>Our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks.
  arXiv  Detail & Related papers  (2026-06-28T12:12:18Z)
• Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructions [61.479946958462754]
  We study multi-task reinforcement learning (RL), a setting in which an agent learns a single, universal policy.<n>We present a novel task embedding technique leveraging a new generation of semantic translations-to-automata.
  arXiv  Detail & Related papers  (2026-02-06T14:46:27Z)
• HELP: Hierarchical Embodied Language Planner for Household Tasks [75.38606213726906]
  Embodied agents tasked with complex scenarios rely heavily on robust planning capabilities.<n>Large language models equipped with extensive linguistic knowledge can play this role.<n>We propose a Hierarchical Embodied Language Planner, called HELP, consisting of a set of LLM-based agents.
  arXiv  Detail & Related papers  (2025-12-25T15:54:08Z)
• Zero-Shot Instruction Following in RL via Structured LTL Representations [54.08661695738909]
  Linear temporal logic (LTL) is a compelling framework for specifying complex, structured tasks for reinforcement learning (RL) agents.<n>Recent work has shown that interpreting instructions as finite automata, which can be seen as high-level programs monitoring task progress, enables learning a single generalist policy capable of executing arbitrary instructions at test time.<n>We propose a novel approach to learning a multi-task policy for following arbitrary instructions that addresses this shortcoming.
  arXiv  Detail & Related papers  (2025-12-02T10:44:51Z)
• LangPert: Detecting and Handling Task-level Perturbations for Robust Object Rearrangement [21.236557779562794]
  LangPert is a language-based framework designed to detect and mitigate Task-Level Perturbations (TLP)<n>LangPert integrates a Visual Language Model (VLM) to comprehensively monitor policy's skill execution and environmental TLP.<n>Our experimental results demonstrate that LangPert handles diverse TLP situations more effectively than baseline methods.
  arXiv  Detail & Related papers  (2025-04-14T05:39:15Z)
• Aligning LLM+PDDL Symbolic Plans with Human Objective Specifications through Evolutionary Algorithm Guidance [1.8974443450771445]
  We develop an evolutionary approach to generate a population of symbolic goal specifications.<n>We evaluate our approach on a collection of prototypical specifications in a notional naval disaster recovery task.
  arXiv  Detail & Related papers  (2024-11-30T00:58:48Z)
• CoT-TL: Low-Resource Temporal Knowledge Representation of Planning Instructions Using Chain-of-Thought Reasoning [1.433758865948252]
  We introduce CoT-TL, a data-efficient in-context learning framework for translating natural language specifications into representations.<n>CoT-TL achieves state-of-the-art accuracy across three diverse datasets in low-data scenarios.<n>We incorporate CoT-TL's practicality into a QuadCopter for multi-step drone planning based on natural language instructions.
  arXiv  Detail & Related papers  (2024-10-21T17:10:43Z)
• Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation [8.180994118420053]
  Nl2Hltl2Plan is a framework that translates natural language commands into hierarchical Linear Temporal Logic (LTL)<n>First, an LLM transforms instructions into a Hierarchical Task Tree, capturing logical and temporal relations.<n>Next, a fine-tuned LLM converts sub-tasks into flat formulas, which are aggregated into hierarchical specifications.
  arXiv  Detail & Related papers  (2024-08-15T14:46:13Z)
• From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control [58.72492647570062]
  We introduce our method -- Learnable Latent Codes as Bridges (LCB) -- as an alternate architecture to overcome limitations.<n>We find that methodoutperforms baselines that leverage pure language as the interface layer on tasks that require reasoning and multi-step behaviors.
  arXiv  Detail & Related papers  (2024-05-08T04:14:06Z)
• Learning adaptive planning representations with natural language guidance [90.24449752926866]
  This paper describes Ada, a framework for automatically constructing task-specific planning representations. Ada interactively learns a library of planner-compatible high-level action abstractions and low-level controllers adapted to a particular domain of planning tasks.
  arXiv  Detail & Related papers  (2023-12-13T23:35:31Z)
• Generalizing to New Domains by Mapping Natural Language to Lifted LTL [20.58567011476273]
  We introduce an intermediate contextual query representation which can be learned from single positive task specification examples. We compare our method to state-of-the-art CopyNet models capable of translating natural language. We demonstrate that our method outputs can be used for planning in a simulated OO-MDP environment.
  arXiv  Detail & Related papers  (2021-10-11T20:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.