NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

• URL: http://arxiv.org/abs/2507.08800v1
• Date: Fri, 11 Jul 2025 17:59:40 GMT
• Title: NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
• Authors: Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng,
• Abstract summary: We introduce NeuralOS, a neural framework that simulates user interfaces (GUIs) of operating systems by directly predicting screen frames.<n>NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural sequences that generates screen images.<n> Experiments show that NeuralOS successfully renders realistic, accurately captures mouse interactions, and reliably predicts state transitions like application launches.
• Score: 49.32972670096748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a large-scale dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Although modeling fine-grained keyboard interactions precisely remains challenging, NeuralOS offers a step toward creating fully adaptive, generative neural interfaces for future human-computer interaction systems.

Related papers

• Unsupervised representation learning with Hebbian synaptic and structural plasticity in brain-like feedforward neural networks [0.0]
  We introduce and evaluate a brain-like neural network model capable of unsupervised representation learning.<n>The model was tested on a diverse set of popular machine learning benchmarks.
  arXiv  Detail & Related papers  (2024-06-07T08:32:30Z)
• Adaptive Robotic Arm Control with a Spiking Recurrent Neural Network on a Digital Accelerator [41.60361484397962]
  We present an overview of the system, and a Python framework to use it on a Pynq ZU platform. We show how the simulated accuracy is preserved with a peak performance of 3.8M events processed per second.
  arXiv  Detail & Related papers  (2024-05-21T14:59:39Z)
• WaLiN-GUI: a graphical and auditory tool for neuron-based encoding [73.88751967207419]
  Neuromorphic computing relies on spike-based, energy-efficient communication. We develop a tool to identify suitable configurations for neuron-based encoding of sample-based data into spike trains. The WaLiN-GUI is provided open source and with documentation.
  arXiv  Detail & Related papers  (2023-10-25T20:34:08Z)
• NeuralClothSim: Neural Deformation Fields Meet the Thin Shell Theory [70.10550467873499]
  We propose NeuralClothSim, a new quasistatic cloth simulator using thin shells. Our memory-efficient solver operates on a new continuous coordinate-based surface representation called neural deformation fields.
  arXiv  Detail & Related papers  (2023-08-24T17:59:54Z)
• Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuits [61.94533459151743]
  This work addresses the challenge of designing neurobiologically-motivated schemes for adjusting the synapses of spiking networks. Our experimental simulations demonstrate a consistent advantage over other biologically-plausible approaches when training recurrent spiking networks.
  arXiv  Detail & Related papers  (2023-03-30T02:40:28Z)
• GenNI: Human-AI Collaboration for Data-Backed Text Generation [102.08127062293111]
  Table2Text systems generate textual output based on structured data utilizing machine learning. GenNI (Generation Negotiation Interface) is an interactive visual system for high-level human-AI collaboration in producing descriptive text.
  arXiv  Detail & Related papers  (2021-10-19T18:07:07Z)
• Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks [0.0]
  Convolutional neural networks (CNNs) are now the de facto solution for computer vision problems. We propose an end-to-end biologically inspired approach using event cameras and spiking neural networks (SNNs) Our method enables the training of sparse spiking neural networks directly on event data, using the popular deep learning framework PyTorch.
  arXiv  Detail & Related papers  (2021-04-26T13:52:01Z)
• Draw your Neural Networks [0.0]
  We present Sketch framework, that uses this GUI-based approach to design and modify the neural networks. The system provides popular layers and operations out-of-the-box and could import any supported pre-trained model.
  arXiv  Detail & Related papers  (2020-12-12T09:44:03Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)
• Nengo and low-power AI hardware for robust, embedded neurorobotics [6.574517227976925]
  We identify four primary challenges in building robust, embedded neurorobotic systems. We present two examples using Nengo to develop neural networks that run on CPU, GPU, and Intel's neuromorphic chip, Loihi.
  arXiv  Detail & Related papers  (2020-07-20T16:17:27Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)
• Self learning robot using real-time neural networks [7.347989843033033]
  This paper involves research, development and experimental analysis of a neural network implemented on a robot with an arm. The neural network learns using the algorithms of Gradient Descent and Backpropagation. Both the implementation and training of the neural network is done locally on the robot on a raspberry pi 3 so that its learning process is completely independent.
  arXiv  Detail & Related papers  (2020-01-06T13:13:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.