Logo image
Back
Estimating Error and Bias in Offline Evaluation Results
Conference proceeding   Open access

Estimating Error and Bias in Offline Evaluation Results

Mucun Tian, Michael D. Ekstrand and Assoc Comp Machinery
CHIIR'20: PROCEEDINGS OF THE 2020 CONFERENCE ON HUMAN INFORMATION INTERACTION AND RETRIEVAL, pp 392-396
01 Jan 2020
DOI: https://doi.org/10.1145/3343413.3378004
url
https://doi.org/10.1145/3343413.3378004View
Published, Version of Record (VoR) Open

Abstract

Computer Science Computer Science, Cybernetics Computer Science, Theory & Methods Science & Technology Technology
Offline evaluations of recommender systems attempt to estimate users' satisfaction with recommendations using static data from prior user interactions. These evaluations provide researchers and developers with first approximations of the likely performance of a new system and help weed out bad ideas before presenting them to users. However, offline evaluation cannot accurately assess novel, relevant recommendations, because the most novel items were previously unknown to the user, so they are missing from the historical data and cannot be judged as relevant. We present a simulation study to estimate the error that such missing data causes in commonly-used evaluation metrics in order to assess its prevalence and impact. We find that missing data in the rating or observation process causes the evaluation protocol to systematically mis-estimate metric values, and in some cases erroneously determine that a popularity-based recommender outperforms even a perfect personalized recommender. Substantial breakthroughs in recommendation quality, therefore, will be difficult to assess with existing offline techniques.

Metrics

15 Record Views
8 citations in Web of Science
7 citations in Scopus

Details

InCites Highlights

Data related to this publication, from InCites Benchmarking & Analytics tool:

Web of Science research areas
Computer Science, Cybernetics
Computer Science, Theory & Methods
Logo image