Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support

• URL: http://arxiv.org/abs/2511.20656v1
• Date: Fri, 10 Oct 2025 10:58:15 GMT
• Title: Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support
• Authors: Haowen Xu, Jose Tupayachi, Xiao-Ying Yu,
• Abstract summary: Development of web-based dashboards for risk analysis and decision making often challenged by difficulty in big, multidimensional data.<n>We introduce a generative AI framework that automates the creation of interactive geospatial dashboards from user-defined inputs.
• Score: 1.506501956463029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The development of web-based geospatial dashboards for risk analysis and decision support is often challenged by the difficulty in visualization of big, multi-dimensional environmental data, implementation complexity, and limited automation. We introduce a generative AI framework that harnesses Large Language Models (LLMs) to automate the creation of interactive geospatial dashboards from user-defined inputs including UI wireframes, requirements, and data sources. By incorporating a structured knowledge graph, the workflow embeds domain knowledge into the generation process and enable accurate and context-aware code completions. A key component of our approach is the Context-Aware Visual Prompting (CAVP) mechanism, which extracts encodes and interface semantics from visual layouts to guide LLM driven generation of codes. The new framework also integrates a self-validation mechanism that uses an agent-based LLM and Pass@k evaluation alongside semantic metrics to assure output reliability. Dashboard snippets are paired with data visualization codebases and ontological representations, enabling a pipeline that produces scalable React-based completions using the MVVM architectural pattern. Our results demonstrate improved performance over baseline approaches and expanded functionality over third party platforms, while incorporating multi-page, fully functional interfaces. We successfully developed a framework to implement LLMs, demonstrated the pipeline for automated code generation, deployment, and performed chain-of-thought AI agents in self-validation. This integrative approach is guided by structured knowledge and visual prompts, providing an innovative geospatial solution in enhancing risk analysis and decision making.

Related papers

• The Auton Agentic AI Framework [5.410458076724158]
  The field of Artificial Intelligence is undergoing a transition from Generative AI to Agentic AI.<n>This transition exposes a fundamental architectural mismatch: Large Language Models (LLMs) produce unstructured outputs, whereas the backend infrastructure they must control requires deterministic, schema-conformant inputs.<n>The present paper describes the Auton Agentic AI Framework, a principled architecture for the creation, creation, and governance of autonomous agent.
  arXiv  Detail & Related papers  (2026-02-27T06:42:08Z)
• El Agente Gráfico: Structured Execution Graphs for Scientific Agents [7.47895130442454]
  We present El Agente Grfico, a single-agent framework that embeds large language models (LLMs)-driven decision-making within a type-safe execution environment.<n>Central to our approach is a structured abstraction of scientific concepts and an object-graph mapper that represents computational state as typed Python objects.<n>We evaluate the system by developing an automated benchmarking framework across a suite of university-level quantum chemistry tasks.
  arXiv  Detail & Related papers  (2026-02-19T23:47:05Z)
• The Agentic Automation Canvas: a structured framework for agentic AI project design [0.0]
  We present the Agentic Automation Canvas (AAC), a structured framework for the prospective design of agentic systems.<n> AAC captures six dimensions of an automation project: definition and scope; user expectations with quantified benefit metrics; developer feasibility assessments; governance staging.<n>It is made accessible through a client-side web application with real-time validation.
  arXiv  Detail & Related papers  (2026-02-16T16:46:04Z)
• AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesis [30.512393568258105]
  Large Language Model agents demonstrate potential in solving real-world problems via tools, yet generalist intelligence is bottlenecked by scarce high-quality, long-horizon data.<n>We propose AgentSkiller, a fully automated framework synthesizing multi-turn interaction data across realistic, semantically linked domains.
  arXiv  Detail & Related papers  (2026-02-10T03:21:42Z)
• AI Agent for Reverse-Engineering Legacy Finite-Difference Code and Translating to Devito [0.0]
  This study develops an integrated AI framework to facilitate the transformation of legacy finite difference implementations into the Devito environment.<n>Retrieval-Augmented Generation (RAG) and open-source Large Language Models are combined through multi-stage iterative in the system's hybrid LangGraph architecture.
  arXiv  Detail & Related papers  (2026-01-26T11:31:00Z)
• Z-Space: A Multi-Agent Tool Orchestration Framework for Enterprise-Grade LLM Automation [3.518072776386001]
  This paper proposes Z-Space, a data-generation-oriented multi-agent collaborative tool invocation framework.<n>The framework has been deployed in the Eleme platform's technical division, serving large-scale test data generation scenarios.<n>Production data demonstrates that the system reduces average token consumption in tool inference by 96.26%.
  arXiv  Detail & Related papers  (2025-11-23T03:59:14Z)
• AUTO-Explorer: Automated Data Collection for GUI Agent [58.58097564914626]
  We propose an automated data collection method with minimal annotation costs, named Auto-Explorer.<n>It incorporates a simple yet effective exploration mechanism that autonomously parses and explores GUI environments.<n>Using the data gathered, we fine-tune a multimodal large language model (MLLM) and establish a GUI element grounding testing set.
  arXiv  Detail & Related papers  (2025-11-09T15:13:45Z)
• Affordance Representation and Recognition for Autonomous Agents [64.39018305018904]
  This paper introduces a pattern language for world modeling from structured data.<n>The DOM Transduction Pattern addresses the challenge of web page complexity.<n>The Hypermedia Affordances Recognition Pattern enables the agent to dynamically enrich its world model.
  arXiv  Detail & Related papers  (2025-10-28T14:27:28Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• Agent0: Leveraging LLM Agents to Discover Multi-value Features from Text for Enhanced Recommendations [0.0]
  Large language models (LLMs) and their associated agent-based frameworks have significantly advanced automated information extraction.<n>This paper presents Agent0, an agent-based system designed to automate information extraction and feature construction from raw, unstructured text.
  arXiv  Detail & Related papers  (2025-07-25T06:45:10Z)
• A Survey on (M)LLM-Based GUI Agents [62.57899977018417]
  Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction.<n>Recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms.<n>This survey identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control.
  arXiv  Detail & Related papers  (2025-03-27T17:58:31Z)
• From PowerPoint UI Sketches to Web-Based Applications: Pattern-Driven Code Generation for GIS Dashboard Development Using Knowledge-Augmented LLMs, Context-Aware Visual Prompting, and the React Framework [1.4367082420201918]
  This paper introduces a knowledgeaugmented code generation framework for complex GIS applications.<n>The framework retrieves software engineering best practices, domain, and advanced technology stacks from a specialized knowledge base.
  arXiv  Detail & Related papers  (2025-02-12T19:59:57Z)
• Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining [67.87810796668981]
  Information-Sensitive Cropping (ISC) and Self-Refining Dual Learning (SRDL)<n>Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations.<n>These improvements translate to significant gains in both web and OS agent downstream tasks.
  arXiv  Detail & Related papers  (2024-12-13T18:40:10Z)
• Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflows [1.4582633500696451]
  LangGraph framework is designed to enhance machine learning through scalability, visualization, and intelligent process optimization.<n>At its core, the framework introduces Agent AI, a pivotal innovation that leverages Spark's distributed computing capabilities.<n>The framework also incorporates large language models through the LangChain ecosystem, enhancing interaction with unstructured data.
  arXiv  Detail & Related papers  (2024-12-02T13:41:38Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.