Logo image
Back
A New Approach to Dimensionality Reduction for Anomaly Detection in Data Traffic
Journal article   Open access   Peer reviewed

A New Approach to Dimensionality Reduction for Anomaly Detection in Data Traffic

Tingshan Huang, Harish Sethu and Nagarajan Kandasamy
IEEE eTransactions on network and service management, v 13(3), pp 651-665
Sep 2016
DOI: https://doi.org/10.1109/TNSM.2016.2597125
url
https://doi.org/10.1109/tnsm.2016.2597125View
Accepted (AM)Open Access (Publisher-Specific),  Open

Abstract

anomaly detection Covariance matrices Dimensionality reduction Feature extraction Monitoring Principal component analysis Real-time systems subspace Temperature measurement Training data
The monitoring and management of high-volume feature-rich traffic in large networks offers significant challenges in storage, transmission, and computational costs. The predominant approach to reducing these costs is based on performing a linear mapping of the data to a low-dimensional subspace such that a certain large percentage of the variance in the data is preserved in the low-dimensional representation. This variance-based subspace approach to dimensionality reduction forces a fixed choice of the number of dimensions, is not responsive to real-time shifts in observed traffic patterns, and is vulnerable to normal traffic spoofing. Based on theoretical insights proved in this paper, we propose a new distance-based approach to dimensionality reduction motivated by the fact that the real-time structural differences between the covariance matrices of the observed and the normal traffic is more relevant to anomaly detection than the structure of the training data alone. Our approach, called the distance-based subspace method, allows a different number of reduced dimensions in different time windows and arrives at only the number of dimensions necessary for effective anomaly detection. We present centralized and distributed versions of our algorithm and, using simulation on real traffic traces, demonstrate the qualitative and quantitative advantages of the distance-based subspace approach.

Metrics

14 Record Views
38 citations in Web of Science
40 citations in Scopus

Details

InCites Highlights

Data related to this publication, from InCites Benchmarking & Analytics tool:

Web of Science research areas
Computer Science, Information Systems
Logo image