Anomaly Detection From Time Series Data For Decision Support

Many organizations adopt information technologies to make intelligent decisions during daily operations. Time series data plays a crucial role in supporting such decision making processes. Though current studies on time-series-based enabled (TDE) decision support provide reasonably well results, the anomaly detection essence underling most of the scenarios and the plenitude of unlabeled data are largely overlooked and left unexplored. We argue that by using normal condition forecasting and unsupervised feature learning, these two important research gaps could be filled. Therefore, we designed a framework for the proposed TDE decision support problem. In order to testify its effectiveness, we carried out two experimental studies on anomaly detection from time series traffic data for transportation management in this thesis and the results show that decision support performance in the context of transportation management was significantly improved.In the first experimental study, we present a hybrid approach to detect anomalies in time series traffic data, namely, Automatic Incident Detection (AID) in transportation systems. In this hybrid approach, we incorporate the normal traffic forecasting component to testify the impact of accurate forecasting technique on TDE decision support. In the context of AID, this hybrid framework combines time series analysis and machine learning techniques also in light of the fault diagnosis theory. The time series component is to forecast the normal traffic for the current time point based on prior (normal) traffic. The machine learning component aims to detect incidents using features of real-time traffic, predicted normal traffic and differences between the two. We validate our approach using a real-world dataset collected in the previous research. The results show that the hybrid approach is able to detect incidents more accurately (higher detection rate) and faster (shorter mean time to detect) under the requirement of a similar false alarm rate as compared with state-of-the-art algorithms. This study lends support to further studies on combining time series analysis with machine learning to address TDE decision support in Intelligent Transportation Systems (ITS).In the second experimental study, we tapped on the potential of using deep learning and unsupervised feature learning techniques to generate higher-level features, intending to further improve the feature generation component in the hybrid framework proposed previously and also fill the gap that plenty of unlabeled data is left unexplored. This advanced feature learning technique is able to delve into the structure of unlabeled traffic data and discover the feature basis. Given that real-time AID requires efficient algorithms in large-scale applications, we select a simple yet effective clustering technique, Spherical K-Means, to generate centroids from unlabeled traffic data for higher-level feature mapping. With centroids learnt in the clustering phase, original feature vector composing of traffic information for several lags was transformed into higher-level features which secure the performance enhancement in the consecutive classifier for detecting incidents. We testify the efficiency of higher-level features with SVM based on1-880data set. Comparison between the results of advanced feature and original ones show the potential of using unsupervised feature learning to automatically generate higher-level features for a better AID predication, given the number of centroids was carefully selected.We discuss contributions and limitations of the proposed framework and two experimental studies in chapter6. We conclude with future directions.