Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models

• URL: http://arxiv.org/abs/2609.11135v1
• Date: Thu, 10 Sep 2026 06:26:02 GMT
• Title: Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models
• Abstract summary: Short-term photovoltaic (PV) power and global horizontal irradiance (GHI) forecasts are essential for effective dispatch, reserve scheduling, and grid operations.<n>We propose SolCloudLLM, an LLM-based multimodal forecasting framework.
• Score: 5.367829077016888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Short-term photovoltaic (PV) power and global horizontal irradiance (GHI) forecasts are essential for effective dispatch, reserve scheduling, and grid operations. At these forecasting horizons, errors are predominantly driven by cloud induced ramps: relying solely on historical numerical data may struggle to anticipate an incoming cloud, making ground-based sky images a crucial complementary physical signal. Furthermore, forecast performance is highly sensitive to location and local observing conditions, creating a strong need for site-specific data that are often scarce. Recently, large language models (LLMs) have demonstrated competitive performance and high data efficiency in time-series forecasting. Despite their success, existing LLM-based forecasting methods remain predominantly unimodal, relying primarily on historical numerical time-series data. Effectively incorporating sky imagery into an LLM-based forecasting framework remains under-explored and an open challenge. In this paper, we propose SolCloudLLM, an LLM-based multimodal forecasting framework. SolCloudLLM aligns sky-image patches with time-series patches and fuses their corresponding representations through bidirectional multimodal fusion, yielding a unified representation that is subsequently mapped into the embedding space of an LLM. Extensive experiments on the SIRTA and SKIPP'D datasets demonstrate that SolCloudLLM consistently outperforms the best baseline methods in MSE across all forecasting horizons, achieving a maximum relative MSE reduction of 25.4%. Stratified analysis further indicates that the benefits of multimodal fusion are concentrated primarily under cloudy conditions. Notably, SolCloudLLM achieves the best performance in nearly all few-shot settings, whereas other deep learning baselines experience substantial performance degradation and are frequently outperformed by the non-learning physical method.

Related papers

• Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning [0.16112718683989877]
  This study examines the ability of large language models to predict the risk of weather-related forced outages in the distribution grid.<n>The problem is formulated as a binary severity classification task using six years of outage records and high-resolution weather data.
  arXiv  Detail & Related papers  (2026-09-02T19:38:37Z)
• FarSky: Task-Aware Latent-Space Coupling for Generative Intra-Hour Solar Forecasting [0.0]
  FarSky is a generative forecasting framework that learns task-aware representations of sky images.<n>It achieves the best overall deterministic and probabilistic forecasting performance, improving forecast skill by up to 11 percentage points.<n>It substantially improves ramp event detection over existing methods, achieving F1-scores above 60%.
  arXiv  Detail & Related papers  (2026-08-06T08:32:56Z)
• Distribution-Aware Diffusion-LLM for Robust Ultra-Long-Term Time Series Forecasting [8.2867621159451]
  Time series forecasting is a fundamental machine learning task.<n>Recent work has explored Large Language Models (LLMs) for this purpose due to their strong generalization, pattern recognition, and zero-shot or few-shot capabilities.<n>We propose a new framework Diffusion-LLM that integrates a conditional diffusion model into an LLM-based forecasting pipeline.
  arXiv  Detail & Related papers  (2026-06-22T14:18:28Z)
• Step-adaptive multimodal fusion network with multi-scale cloud feature learning for ultra-short-term solar irradiance forecasting [0.0]
  This paper proposes a multi-source data fusion model for ultra-short-term irradiance prediction.<n>The model first employs InceptionNeXt to extract multi-scale, multi-directional spatial features from ground-based cloud images.<n>A step-adaptive low-frequency compensation unit is then introduced to dynamically modulate global low-frequency information based on the prediction step.
  arXiv  Detail & Related papers  (2026-06-04T12:42:52Z)
• Nexus : An Agentic Framework for Time Series Forecasting [72.73790673303154]
  Time series forecasting requires reasoning with unstructured contextual data such as news or events.<n>We introduce Nexus, a multi-agent forecasting framework that decomposes prediction into specialized stages.<n>We show that current-generation LLMs possess substantially stronger intrinsic forecasting ability than previously recognized.
  arXiv  Detail & Related papers  (2026-05-14T05:12:13Z)
• LEAF: A Living Benchmark for Event-Augmented Forecasting [78.94019460534442]
  LEAF is the first living benchmark for event-augmented forecasting tasks.<n>We evaluate state-of-the-art proprietary and open-weight LLMs.<n>We find that LLMs achieve better performance on equities they confidently identify as more predictable.
  arXiv  Detail & Related papers  (2026-05-09T03:17:59Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• Vision-LLMs for Spatiotemporal Traffic Forecasting [14.700408329373998]
  Large Language Models (LLMs) inherently struggle to model the complex spatial dependencies of grid-based traffic data.<n>We propose ST-Vision-LLM, a novel framework reframe thatstemporal forecasting as a vision-language fusion problem.<n>We show that ST-Vision-LLM outperforms existing methods by 15.6% in long-term prediction accuracy and exceeds the second-best baseline by over 30.04% in cross-domain scenarios.
  arXiv  Detail & Related papers  (2025-10-13T11:15:56Z)
• LLM-PS: Empowering Large Language Models for Time Series Forecasting with Temporal Patterns and Semantics [56.99021951927683]
  Time Series Forecasting (TSF) is critical in many real-world domains like financial planning and health monitoring.<n>Existing Large Language Models (LLMs) usually perform suboptimally because they neglect the inherent characteristics of time series data.<n>We propose LLM-PS to empower the LLM for TSF by learning the fundamental textitPatterns and meaningful textitSemantics from time series data.
  arXiv  Detail & Related papers  (2025-03-12T11:45:11Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context.<n>We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters.<n>We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• UniMix: Towards Domain Adaptive and Generalizable LiDAR Semantic Segmentation in Adverse Weather [55.95708988160047]
  LiDAR semantic segmentation (LSS) is a critical task in autonomous driving. Prior LSS methods are investigated and evaluated on datasets within the same domain in clear weather. We propose UniMix, a universal method that enhances the adaptability and generalizability of LSS models.
  arXiv  Detail & Related papers  (2024-04-08T02:02:15Z)
• Contextualizing MLP-Mixers Spatiotemporally for Urban Data Forecast at Scale [54.15522908057831]
  We propose an adapted version of the computationally-Mixer for STTD forecast at scale. Our results surprisingly show that this simple-yeteffective solution can rival SOTA baselines when tested on several traffic benchmarks. Our findings contribute to the exploration of simple-yet-effective models for real-world STTD forecasting.
  arXiv  Detail & Related papers  (2023-07-04T05:19:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.