Affordance Representation and Recognition for Autonomous Agents

• URL: http://arxiv.org/abs/2510.24459v1
• Date: Tue, 28 Oct 2025 14:27:28 GMT
• Title: Affordance Representation and Recognition for Autonomous Agents
• Authors: Habtom Kahsay Gidey, Niklas Huber, Alexander Lenz, Alois Knoll,
• Abstract summary: This paper introduces a pattern language for world modeling from structured data.<n>The DOM Transduction Pattern addresses the challenge of web page complexity.<n>The Hypermedia Affordances Recognition Pattern enables the agent to dynamically enrich its world model.
• Score: 64.39018305018904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The autonomy of software agents is fundamentally dependent on their ability to construct an actionable internal world model from the structured data that defines their digital environment, such as the Document Object Model (DOM) of web pages and the semantic descriptions of web services. However, constructing this world model from raw structured data presents two critical challenges: the verbosity of raw HTML makes it computationally intractable for direct use by foundation models, while the static nature of hardcoded API integrations prevents agents from adapting to evolving services. This paper introduces a pattern language for world modeling from structured data, presenting two complementary architectural patterns. The DOM Transduction Pattern addresses the challenge of web page complexity by distilling} a verbose, raw DOM into a compact, task-relevant representation or world model optimized for an agent's reasoning core. Concurrently, the Hypermedia Affordances Recognition Pattern enables the agent to dynamically enrich its world model by parsing standardized semantic descriptions to discover and integrate the capabilities of unknown web services at runtime. Together, these patterns provide a robust framework for engineering agents that can efficiently construct and maintain an accurate world model, enabling scalable, adaptive, and interoperable automation across the web and its extended resources.

Related papers

• Web World Models [60.208836336654315]
  We introduce the Web World Model (WWM), a middle ground where world state and physics'' are implemented in ordinary web code.<n>We build a suite of WWMs on a realistic web stack, including an infinite travel atlas grounded in real geography, fictional galaxy explorers, web-scale encyclopedic and narrative worlds, and simulation- and game-like environments.<n>Our results suggest that web stacks themselves can serve as a scalable substrate for world models, enabling controllable yet open-ended environments.
  arXiv  Detail & Related papers  (2025-12-29T18:31:45Z)
• From Word to World: Can Large Language Models be Implicit Text-based World Models? [82.47317196099907]
  Agentic reinforcement learning increasingly relies on experience-driven scaling.<n>World models offer a potential way to improve learning efficiency through simulated experience.<n>We study whether large language models can reliably serve this role and under what conditions they meaningfully benefit agents.
  arXiv  Detail & Related papers  (2025-12-21T17:28:42Z)
• Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support [1.506501956463029]
  Development of web-based dashboards for risk analysis and decision making often challenged by difficulty in big, multidimensional data.<n>We introduce a generative AI framework that automates the creation of interactive geospatial dashboards from user-defined inputs.
  arXiv  Detail & Related papers  (2025-10-10T10:58:15Z)
• World Model Implanting for Test-time Adaptation of Embodied Agents [29.514831254621438]
  In embodied AI, a persistent challenge is enabling agents to robustly adapt to novel domains without requiring extensive data collection or retraining.<n>We present a world model implanting framework (WorMI) that combines the reasoning capabilities of large language models with independently learned, domain-specific world models.<n>We evaluate our WorMI on the VirtualHome and ALFWorld benchmarks, demonstrating superior zero-shot and few-shot performance compared to several LLM-based approaches.
  arXiv  Detail & Related papers  (2025-09-04T07:32:16Z)
• AI-assisted JSON Schema Creation and Mapping [0.0]
  We present a hybrid approach that combines large language models (LLMs) with deterministic techniques to enable creation, modification, and schema mapping based on natural language inputs by the user.<n>This work significantly lowers the barrier to structured data modeling and data integration for non-experts.
  arXiv  Detail & Related papers  (2025-08-07T09:27:10Z)
• WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization [68.46693401421923]
  WebShaper systematically formalizes IS tasks through set theory.<n>WebShaper achieves state-of-the-art performance among open-sourced IS agents on GAIA and WebWalkerQA benchmarks.
  arXiv  Detail & Related papers  (2025-07-20T17:53:37Z)
• WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis [34.998277998052444]
  We propose WebSynthesis, a novel framework for trajectory synthesis and training.<n>We show that an agent trained using WebSynthesis on a small-scale synthetic dataset achieves performance comparable to or even surpassing that of models trained on large-scale real-world data.
  arXiv  Detail & Related papers  (2025-07-06T12:31:10Z)
• World Models for Cognitive Agents: Transforming Edge Intelligence in Future Networks [55.90051810762702]
  We present a comprehensive overview of world models, highlighting their architecture, training paradigms, and applications across prediction, generation, planning, and causal reasoning.<n>We propose Wireless Dreamer, a novel world model-based reinforcement learning framework tailored for wireless edge intelligence optimization.
  arXiv  Detail & Related papers  (2025-05-31T06:43:00Z)
• RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs [3.41612427812159]
  In digital content creation tools, users express their needs through natural language queries that must be mapped to API calls.<n>Existing approaches to synthetic data generation fail to replicate real-world data distributions.<n>We present a novel router-based architecture that generates high-quality synthetic training data.
  arXiv  Detail & Related papers  (2025-05-15T16:53:45Z)
• WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model [55.276852838877346]
  Self-evolving agents are trained on trajectories sampled autonomously based on their own policies.<n>We propose a novel framework that introduces a co-evolving World Model LLM.<n>This world model predicts the next observation based on the current observation and action within the web environment.
  arXiv  Detail & Related papers  (2025-04-23T02:54:31Z)
• Agent-driven Generative Semantic Communication with Cross-Modality and Prediction [57.335922373309074]
  We propose a novel agent-driven generative semantic communication framework based on reinforcement learning. In this work, we develop an agent-assisted semantic encoder with cross-modality capability, which can track the semantic changes, channel condition, to perform adaptive semantic extraction and sampling. The effectiveness of the designed models has been verified using the UA-DETRAC dataset, demonstrating the performance gains of the overall A-GSC framework.
  arXiv  Detail & Related papers  (2024-04-10T13:24:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.