Journal article
RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures
Proceedings of the VLDB Endowment, v 4(11), pp 911-920
01 Aug 2011
Abstract
Traditionally, recommender systems have been "hand-built", implemented as custom applications hard-wired to a particular recommendation task. Recently, the database community has begun exploring alternative DBMS-based recommender system architectures, whereby a database both stores the recommender system data (e.g., ratings data and the derived recommender models) and generates recommendations using SQL queries. In this paper, we present a comprehensive experimental comparison of both architectures. We define a set of benchmark tasks based on the needs of a typical recommender-powered e-commerce site. We then evaluate the performance of the "hand-built" MultiLens recommender application against two DBMS-based implementations: an unmodified DBMS and RecStore, a DBMS modified to improve efficiency in incremental recommender model updates. We employ two non-trivial data sets in our study: the 10 million rating MovieLens data, and the 100 million rating data set used in the Netflix Challenge. This study is the first of its kind, and our findings reveal an interesting trade-off: "hand-built" recommenders exhibit superior performance in model-building and pure recommendation tasks, while DBMS-based recommenders are superior at more complex recommendation tasks such as providing filtered recommendations and blending text-search with recommendation prediction scores.
Metrics
Details
- Title
- RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures
- Creators
- Justin J. Levandoski - University of Minnesota SystemMichael D. Ekstrand - University of Minnesota SystemMichael J. Ludwig - University of Minnesota SystemAhmed Eldawy - University of Minnesota SystemMohamed F. Mokbel - University of Minnesota SystemJohn T. Riedl - University of Minnesota System
- Publication Details
- Proceedings of the VLDB Endowment, v 4(11), pp 911-920
- Publisher
- Assoc Computing Machinery
- Number of pages
- 10
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Information Science
- Web of Science ID
- WOS:000219699700020
- Scopus ID
- 2-s2.0-84863795647
- Other Identifier
- 991021868725004721
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Web of Science research areas
- Computer Science, Information Systems
- Computer Science, Theory & Methods