Solaris: Towards Interfaces That Are Generated, Not Coded

• URL: http://arxiv.org/abs/2609.00776v1
• Date: Tue, 01 Sep 2026 06:10:49 GMT
• Title: Solaris: Towards Interfaces That Are Generated, Not Coded
• Abstract summary: We introduce Solaris, an interface world model that generates an interactive UI directly, frame by frame, in response to user actions.<n> Solaris treats mouse interactions as conditioning signals and autoregressively synthesizes the resulting visual state at interactive speeds.<n>A language model complements the visual world model by interpreting user intent and specifying how interactions should affect the generated environment.
• Score: 61.80943145058668
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Digital interfaces are traditionally implemented through intermediate representations such as code, requiring their appearance and behavior to be specified in advance. We introduce Solaris, an interface world model that instead generates an interactive UI directly, frame by frame, in response to user actions. Solaris treats mouse interactions as conditioning signals and autoregressively synthesizes the resulting visual state at interactive speeds. To enable real-time generation while maintaining visual coherence over extended interactions, we combine autoregressive frame generation with few-step distillation and training on the model's own outputs. A language model complements the visual world model by interpreting user intent and specifying how interactions should affect the generated environment, separating high-level reasoning from visual rendering. By generating both the appearance and behavior of an interface dynamically, Solaris enables open-ended interactions that need not be explicitly programmed in advance. We view interface world models as a step toward a new paradigm for software, where interfaces are generated and adapted continuously around user intent rather than implemented as fixed collections of predefined states and

Related papers

• Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages [22.804937497354754]
  We introduce WebIGBench, the first benchmark to evaluate code generation for interactive webpages with complex interactions.<n>By combining manually designed interaction paths with UI automation, we collected 103 complex webpages from real-world websites.<n>This benchmark covers 5 popular interactive action types (e.g., click, input) involving 871 distinct interactive actions.
  arXiv  Detail & Related papers  (2026-05-29T05:45:50Z)
• From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks [5.208702297063032]
  We transform static visual in-context learners into user-driven systems, i.e., Interactive DeLVM.<n>Our work bridges the gap between rigid static task adaptation and fluid interactivity for user-centric visual in-context learning.
  arXiv  Detail & Related papers  (2026-04-08T07:13:56Z)
• MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines [55.641812280108574]
  We introduce an explicit external memory into the system that is continually updated by user actions and queried throughout the generation roll-out.<n>This design gives users direct, editable control over environment structure via an editable memory representation.<n>It naturally extends to real-time multiplayer rollouts with coherent viewpoints and consistent cross-player interactions.
  arXiv  Detail & Related papers  (2026-03-03T18:58:17Z)
• Generative Interfaces for Language Models [70.25765232527762]
  We propose a paradigm in which large language models (LLMs) respond to user queries by proactively generating user interfaces (UIs)<n>Our framework leverages structured interface-specific representations and iterative refinements to translate user queries into task-specific UIs.<n>Results show that generative interfaces consistently outperform conversational ones, with up to a 72% improvement in human preference.
  arXiv  Detail & Related papers  (2025-08-26T17:43:20Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  Aguvis is a vision-based framework for autonomous GUI agents.<n>It standardizes cross-platform interactions and incorporates structured reasoning via inner monologue.<n>It achieves state-of-the-art performance across offline and real-world online benchmarks.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)
• InterFormer: Real-time Interactive Image Segmentation [80.45763765116175]
  Interactive image segmentation enables annotators to efficiently perform pixel-level annotation for segmentation tasks. The existing interactive segmentation pipeline suffers from inefficient computations of interactive models. We propose a method named InterFormer that follows a new pipeline to address these issues.
  arXiv  Detail & Related papers  (2023-04-06T08:57:00Z)
• NL2INTERFACE: Interactive Visualization Interface Generation from Natural Language Queries [19.355412315639462]
  NL2INTERFACE generates interactive multi-visualization interfaces from natural language queries. Users can interact with the interfaces to easily transform the data and quickly see the results in the visualizations.
  arXiv  Detail & Related papers  (2022-09-19T08:31:50Z)
• VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction [50.986371459817256]
  We propose a novel textitVirtual InteRacTion mechanism, termed as VIRT, to enable full and deep interaction modeling in representation-based models. VIRT asks representation-based encoders to conduct virtual interactions to mimic the behaviors as interaction-based models do.
  arXiv  Detail & Related papers  (2021-12-08T09:49:28Z)
• Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion [68.45737688496654]
  We present a modular interactive VOS framework which decouples interaction-to-mask and mask propagation. We show that our method outperforms current state-of-the-art algorithms while requiring fewer frame interactions.
  arXiv  Detail & Related papers  (2021-03-14T14:39:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.