Abstract: Discrete hidden Markov models (HMM) are often applied to malware detection
and classification problems. However, the continuous analog of discrete HMMs,
that is, Gaussian mixture model-HMMs (GMM-HMM), are rarely considered in the
field of cybersecurity. In this paper, we use GMM-HMMs for malware
classification and we compare our results to those obtained using discrete
HMMs. As features, we consider opcode sequences and entropy-based sequences.
For our opcode features, GMM-HMMs produce results that are comparable to those
obtained using discrete HMMs, whereas for our entropy-based features, GMM-HMMs
generally improve significantly on the classification results that we have
achieved with discrete HMMs.