Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

• URL: http://arxiv.org/abs/2603.08930v1
• Date: Mon, 09 Mar 2026 20:58:43 GMT
• Title: Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning
• Abstract summary: This paper introduces a benchmark to evaluate the performance of vision language models (VLMs) in generating plant simulation for digital twins.<n>We propose a novel approach that leverages state-of-the-art open-source VLMs to directly generate simulation parameters in format from drone-based remote sensing images.
• Score: 6.254251081017878
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces a synthetic benchmark to evaluate the performance of vision language models (VLMs) in generating plant simulation configurations for digital twins. While functional-structural plant models (FSPMs) are useful tools for simulating biophysical processes in agricultural environments, their high complexity and low throughput create bottlenecks for deployment at scale. We propose a novel approach that leverages state-of-the-art open-source VLMs -- Gemma 3 and Qwen3-VL -- to directly generate simulation parameters in JSON format from drone-based remote sensing images. Using a synthetic cowpea plot dataset generated via the Helios 3D procedural plant generation library, we tested five in-context learning methods and evaluated the models across three categories: JSON integrity, geometric evaluations, and biophysical evaluations. Our results show that while VLMs can interpret structural metadata and estimate parameters like plant count and sun azimuth, they often exhibit performance degradation due to contextual bias or rely on dataset means when visual cues are insufficient. Validation on a real-world drone orthophoto dataset and an ablation study using a blind baseline further characterize the models' reasoning capabilities versus their reliance on contextual priors. To the best of our knowledge, this is the first study to utilize VLMs to generate structural JSON configurations for plant simulations, providing a scalable framework for reconstruction 3D plots for digital twin in agriculture.

Related papers

• SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models [88.64689505355055]
  We introduce SolarWM, a fully open foundation for building interactive video world models from data preparation through long-horizon inference.<n>SolarWM addresses this coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework.<n>We instantiate four 5B--33B models based on Wan2.2, LTX-2.5, and MiniMax-H3 while preserving their native representations and objectives.<n>The resulting causal models enable real-time interaction over rollouts ranging from minutes to hours after being trained on only 5s sequences.
  arXiv  Detail & Related papers  (2026-09-02T17:59:41Z)
• Text-conditioned Segmentation for Tomato Phenotyping via Procedural Synthetic Data [15.061363878701831]
  Vision-based automation is an excellent candidate for reducing manual labor in greenhouse crop production.<n>Recent advances in vision-based foundational models have shown promising results in zero-shot generalization to novel domains.<n>We present a framework for tomato plant segmentation that combines synthetic data generation with fine-tuning of a foundation model.
  arXiv  Detail & Related papers  (2026-07-20T23:18:55Z)
• Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation [62.51953630639423]
  Existing benchmarks lack visual realism, creating a large domain gap between simulation and reality.<n>We propose VISER, a visually realistic benchmark for evaluating robot manipulation in simulation.<n>VISER features a high-fidelity dataset of over 1,000 3D assets with physically-based rendering (PBR) materials, along with 3D scenes created from these assets through curated layouts or generation.
  arXiv  Detail & Related papers  (2026-05-07T14:13:05Z)
• Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation [63.01022057888141]
  Asset Harvester is an image-to-3D model and end-to-end pipeline that converts sparse, in-thewild object observations from real driving logs into complete, simulation-ready assets.<n>SparseViewDiT is explicitly designed to address limited-angle views and other real-world data challenges.
  arXiv  Detail & Related papers  (2026-04-20T16:20:57Z)
• LLM-Driven 3D Scene Generation of Agricultural Simulation Environments [1.002902747701998]
  Large Language Models (LLMs) for 3D scene generation show promise but often lack domain-specific reasoning, verification mechanisms, and modular design.<n>This paper investigates the use of LLMs to generate agricultural synthetic simulation environments from natural language prompts.<n>A modular multi-LLM pipeline was developed, integrating 3D asset retrieval, domain knowledge injection, and code generation for the Unreal rendering engine.
  arXiv  Detail & Related papers  (2026-02-12T08:33:01Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• FloraForge: LLM-Assisted Procedural Generation of Editable and Analysis-Ready 3D Plant Geometric Models For Agricultural Applications [13.923496304391044]
  We present FloraForge, an LLM-assisted framework that enables domain experts to generate biologically accurate, fully parametric 3D plant models.<n>Our framework leverages LLM-enabled co-design to refine Python scripts that generate parameterized plant as hierarchical B-spline surface representations.<n>We demonstrate the framework on maize, soybean, and mung bean, fitting procedural models to empirical point cloud data.
  arXiv  Detail & Related papers  (2025-12-11T23:28:25Z)
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model [76.08429266631823]
  We propose an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM)<n>URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction.<n> Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-11-02T13:45:51Z)
• Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data [53.040873127309766]
  We propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning.<n>Our method outperforms existing models on both in-dataset and cross-dataset evaluations.
  arXiv  Detail & Related papers  (2025-09-08T17:58:06Z)
• Towards scalable organ level 3D plant segmentation: Bridging the data algorithm computing gap [9.655034225644847]
  Plant phenotyping provides valuable insights into plant environment interactions and genetic evolution.<n>Despite progress in general 3D computer vision domains, the adoption of 3D segmentation for plant phenotyping remains limited by three major challenges.<n>This study bridges the gap between algorithmic advances and practical deployment.
  arXiv  Detail & Related papers  (2025-09-08T04:21:27Z)
• DATR: Diffusion-based 3D Apple Tree Reconstruction Framework with Sparse-View [8.493134772085233]
  This study developed a two-stage framework for the reconstruction of apple trees from sparse views.<n>The framework was evaluated on both field and synthetic datasets.
  arXiv  Detail & Related papers  (2025-08-27T01:45:54Z)
• A large-scale, physically-based synthetic dataset for satellite pose estimation [0.0]
  This paper introduces the DLVS3-HST-V1 dataset, which focuses on the Hubble Space Telescope (HST) as a complex, articulated target.<n>The dataset is generated using advanced real-time and offline rendering technologies, integrating high-fidelity 3D models, dynamic lighting, and physically accurate material properties.<n>The pipeline supports the creation of large-scale, richly annotated image sets with ground-truth 6-DoF pose and keypoint data, semantic segmentation, depth, and normal maps.
  arXiv  Detail & Related papers  (2025-06-15T09:24:32Z)
• BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation [57.40024206484446]
  We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and assets to generate fully customized synthetic data for systematic evaluation of computer vision models. BVS supports a large number of adjustable parameters at the scene level. We showcase three example application scenarios.
  arXiv  Detail & Related papers  (2024-05-15T17:57:56Z)
• Learning from Synthetic Data for Visual Grounding [55.21937116752679]
  We show that SynGround can improve the localization capabilities of off-the-shelf vision-and-language models.<n>Data generated with SynGround improves the pointing game accuracy of a pretrained ALBEF and BLIP models by 4.81% and 17.11% absolute percentage points, respectively.
  arXiv  Detail & Related papers  (2024-03-20T17:59:43Z)
• Unsupervised Learning of 3D Object Categories from Videos in the Wild [75.09720013151247]
  We focus on learning a model from multiple views of a large collection of object instances. We propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T17:57:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.