Online Anomaly Detection with Uncertainty Estimation and Concept Drift Adaptation using Quantile Regression

Abstract

Deep learning algorithms played an important role in big data applications that open other opportunities to study their applicability in anomaly detection. These algorithms are used in prediction-based anomaly detection methods for detection of anomalies in time series data. Various Recurrent Neural Networks (RNN) structures particularly based on Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) have been reported to be used for time series anomaly detection. Despite the performance of these methods, they are affected using a fixed threshold, and the assumption of Gaussian distribution on the prediction error to identify anomalous values. In addition, these techniques do not consider uncertainty in their predictions that may lead to over-confident predictions especially when there is limited training data. This impression motivates our previous research work that proposed a new anomaly detection method called Deep Quantile Regression Anomaly Detection (DQR-AD) that used confidence interval to identify anomalies in time series. However, the speed at which the time series data arrives and the dynamic change of normal behavior in a non-stationary environment will affect the offline training of DQR-AD using historical data. To mitigate these problems, this paper proposed an online DQR-AD that will enable the adaptation of concept changes in the data. Experiments conducted indicate that online DQR-AD method has better performance than its counterpart methods with relatively 10% margin. This result demonstrates how concept drift adaptation strategies adopted in the proposed method improve the performance of anomaly detection in time series.