Files and links (1)
Published, Version of Record (VoR)Open Access (License Unspecified), Open
Conference proceeding
Evaluating Stochastic Rankings with Expected Exposure
CIKM '20: PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON INFORMATION & KNOWLEDGE MANAGEMENT, pp 275-284
01 Jan 2020
Abstract
We introduce the concept of expected exposure as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a distribution over rankings instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including ad hoc retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress.
Metrics
Details
- Title
- Evaluating Stochastic Rankings with Expected Exposure
- Creators
- Fernando Diaz - Microsoft, Montreal, PQ, CanadaBhaskar Mitra - Microsoft Research (United Kingdom)Michael D. Ekstrand - Boise State UniversityAsia J. Biega - Microsoft (Canada)Ben Carterette - Photon Spot (United States)ASSOC COMP MACHINERY
- Publication Details
- CIKM '20: PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON INFORMATION & KNOWLEDGE MANAGEMENT, pp 275-284
- Publisher
- Assoc Computing Machinery
- Number of pages
- 10
- Grant note
- IIS 17-51278 / National Science Foundation; National Science Foundation (NSF)
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Information Science
- Web of Science ID
- WOS:000749561300030
- Scopus ID
- 2-s2.0-85095863065
- Other Identifier
- 991021818388204721
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- International collaboration
- Web of Science research areas
- Computer Science, Information Systems
- Computer Science, Theory & Methods