Logo image
Back
Hash-Based Stream LDA: Topic Modeling in Social Streams
Book chapter   Peer reviewed

Hash-Based Stream LDA: Topic Modeling in Social Streams

Anton Slutsky, Xiaohua Hu and Yuan An
Advances in Knowledge Discovery and Data Mining, pp 151-162
2014
DOI: https://doi.org/10.1007/978-3-319-06608-0_13

Abstract

Generative Probabilistic Model Latent Dirichlet Allocation Random Projection Topic Discovery Topic Modeling
We study the problem of topic modeling in continuous social media streams and propose a new generative probabilistic model called Hash-Based Stream LDA (HS-LDA), which is a generalization of the popular LDA approach. The model differs from LDA in that it exposes facilities to include inter-document similarity in topic modeling. The corresponding inference algorithm outlined in the paper relies on efficient estimation of document similarity with Locality Sensitive Hashing to retain the knowledge of past social discourse in a scalable way. The historical knowledge of previous messages is used in inference to improve quality of topic discovery. Performance of the new algorithm was evaluated against classical LDA approach as well as the stream-oriented On-line LDA and SparseLDA using data sets collected from the Twitter microblog system and an IRC chat community. Experimental results showed that HS-LDA outperformed other techniques by more than 12% for the Twitter dataset and by 21% for the IRC data in terms of average perplexity.

Metrics

12 Record Views
3 citations in Scopus

Details

Logo image