Logo image
Back
Fast botnet detection from streaming logs using online lanczos method
Conference proceeding   Open access

Fast botnet detection from streaming logs using online lanczos method

Zheng Chen, Xinli Yu, Chi Zhang, Jin Zhang, Cui Lin, Bo Song, Jianliang Gao, Xiaohua Hu, Wei-Shih Yang, Erjia Yan, …
2017 IEEE International Conference on Big Data (Big Data), v 2018-, pp 1408-1417
Dec 2017
DOI: https://doi.org/10.1109/BigData.2017.8258074
url
http://arxiv.org/abs/1812.07810View
Submitted Open Open Access (License Unspecified)

Abstract

Correlation Eigenvalues and eigenfunctions Monitoring Principal component analysis Time complexity
Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, real-time protection capability, ease of use and better security of sensitive data. Our algorithm is inspired by a Principal Component Analysis (PCA) to capture correlation in data, and we are first to recognize and adapt Lanczos method to improve the time complexity of PCA-based botnet detection from cubic to sub-cubic, which enables us to more accurately and sensitively detect botnets with sliding time windows rather than fixed time windows. We contribute a generalized online correlation matrix update formula, and a new termination condition for Lanczos iteration for our purpose based on error bound and non-decreasing eigenvalues of symmetric matrices. On our dataset of an ecommerce website logs, experiments show the time cost of Lanczos method with different time windows are consistently only 20% to 25% of PCA.

Metrics

27 Record Views
5 citations in Scopus

Details

Logo image