RLTrace: Synthesizing High-Quality System Call Traces for OS Fuzz Testing

• URL: http://arxiv.org/abs/2310.02609v1
• Date: Wed, 4 Oct 2023 06:46:00 GMT
• Title: RLTrace: Synthesizing High-Quality System Call Traces for OS Fuzz Testing
• Authors: Wei Chen, Huaijin Wang, Weixi Gu, Shuai Wang,
• Abstract summary: We propose a deep reinforcement learning-based solution, called RLTrace, to synthesize diverse and comprehensive system call traces as the seed to fuzz OS kernels. During model training, the deep learning model interacts with OS kernels and infers optimal system call traces. Our evaluation shows that RLTrace outperforms other seed generators by producing more comprehensive system call traces.
• Score: 10.644829779197341
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Securing operating system (OS) kernel is one central challenge in today's cyber security landscape. The cutting-edge testing technique of OS kernel is software fuzz testing. By mutating the program inputs with random variations for iterations, fuzz testing aims to trigger program crashes and hangs caused by potential bugs that can be abused by the inputs. To achieve high OS code coverage, the de facto OS fuzzer typically composes system call traces as the input seed to mutate and to interact with OS kernels. Hence, quality and diversity of the employed system call traces become the prominent factor to decide the effectiveness of OS fuzzing. However, these system call traces to date are generated with hand-coded rules, or by analyzing system call logs of OS utility programs. Our observation shows that such system call traces can only subsume common usage scenarios of OS system calls, and likely omit hidden bugs. In this research, we propose a deep reinforcement learning-based solution, called RLTrace, to synthesize diverse and comprehensive system call traces as the seed to fuzz OS kernels. During model training, the deep learning model interacts with OS kernels and infers optimal system call traces w.r.t. our learning goal -- maximizing kernel code coverage. Our evaluation shows that RLTrace outperforms other seed generators by producing more comprehensive system call traces, subsuming system call corner usage cases and subtle dependencies. By feeding the de facto OS fuzzer, SYZKALLER, with system call traces synthesized by RLTrace, we show that SYZKALLER can achieve higher code coverage for testing Linux kernels. Furthermore, RLTrace found one vulnerability in the Linux kernel (version 5.5-rc6), which is publicly unknown to the best of our knowledge by the time of writing.

Related papers

• KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution [59.20933707301566]
  Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym and kBench.
  arXiv  Detail & Related papers  (2024-07-02T21:44:22Z)
• RelayAttention for Efficient Large Language Model Serving with Long System Prompts [59.50256661158862]
  This paper aims to improve the efficiency of LLM services that involve long system prompts. handling these system prompts requires heavily redundant memory accesses in existing causal attention algorithms. We propose RelayAttention, an attention algorithm that allows reading hidden states from DRAM exactly once for a batch of input tokens.
  arXiv  Detail & Related papers  (2024-02-22T18:58:28Z)
• KernelGPT: Enhanced Kernel Fuzzing via Large Language Models [9.860752730040709]
  We propose KernelGPT, the first approach to automatically inferring Syzkaller specifications via Large Language Models. Our preliminary results demonstrate that KernelGPT can help Syzkaller achieve higher coverage and find multiple previously unknown bugs.
  arXiv  Detail & Related papers  (2023-12-31T18:47:33Z)
• Language Models for Novelty Detection in System Call Traces [0.27309692684728604]
  This paper introduces a novelty detection methodology that relies on a probability distribution over sequences of system calls. The proposed methodology requires minimal expert hand-crafting and achieves an F-score and AuROC greater than 95% on most novelties. The source code and trained models are publicly available on GitHub while the datasets are available on Zenodo.
  arXiv  Detail & Related papers  (2023-09-05T13:11:40Z)
• model-based script synthesis for fuzzing [10.739464605434977]
  Existing approaches fuzz the kernel by modeling syscall sequences from traces or static analysis of system codes. We propose WinkFuzz, an approach to learn and mutate traced syscall sequences in order to reach different kernel states.
  arXiv  Detail & Related papers  (2023-08-08T08:07:50Z)
• SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency [55.49469003537601]
  We introduce SysNoise, a frequently occurred but often overlooked noise in the deep learning training-deployment cycle. We measure the impact of SysNoise on 20+ models, comprehending image classification, object detection, instance segmentation and natural language processing tasks. Our experiments revealed that SysNoise could bring certain impacts on model robustness across different tasks and common mitigations like data augmentation and adversarial training show limited effects on it.
  arXiv  Detail & Related papers  (2023-07-01T09:22:54Z)
• Interactive System-wise Anomaly Detection [66.3766756452743]
  Anomaly detection plays a fundamental role in various applications. It is challenging for existing methods to handle the scenarios where the instances are systems whose characteristics are not readily observed as data. We develop an end-to-end approach which includes an encoder-decoder module that learns system embeddings.
  arXiv  Detail & Related papers  (2023-04-21T02:20:24Z)
• Lifelong Bandit Optimization: No Prior and No Regret [70.94238868711952]
  We develop LIBO, an algorithm which adapts to the environment by learning from past experience. We assume a kernelized structure where the kernel is unknown but shared across all tasks. Our algorithm can be paired with any kernelized or linear bandit algorithm and guarantees optimal performance.
  arXiv  Detail & Related papers  (2022-10-27T14:48:49Z)
• Kernel Continual Learning [117.79080100313722]
  kernel continual learning is a simple but effective variant of continual learning to tackle catastrophic forgetting. episodic memory unit stores a subset of samples for each task to learn task-specific classifiers based on kernel ridge regression. variational random features to learn a data-driven kernel for each task.
  arXiv  Detail & Related papers  (2021-07-12T22:09:30Z)
• Hybrid Deep Neural Networks to Infer State Models of Black-Box Systems [2.294541416972175]
  We propose a hybrid deep neural network that accepts as input a set of time series, one per input/output signal of the system. We have applied our approach on a real UAV auto-pilot solution from our industry partner with half a million lines of C code.
  arXiv  Detail & Related papers  (2020-08-26T23:24:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.