Files and links (1)
Published, Version of Record (VoR)CC BY V4.0, Open
Conference proceeding
Estimation of Fair Ranking Metrics with Incomplete Judgments
PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE 2021 (WWW 2021), pp 1065-1075
01 Jan 2021
Abstract
There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identified using protected attributes such as gender or ethnicity. To date, these metrics typically assume the availability and completeness of protected attribute labels of items. However, the protected attributes of individuals are rarely present, limiting the application of fair ranking metrics in large scale systems. In order to address this problem, we propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items. We evaluate our approach using both simulated and real world data. Our experimental results demonstrate that our method can estimate this family of fair ranking metrics and provides a robust, reliable alternative to exhaustive or random data annotation.
Metrics
Details
- Title
- Estimation of Fair Ranking Metrics with Incomplete Judgments
- Creators
- Omer Kirnap - UCL, London, EnglandFernando Diaz - Microsoft Res, Montreal, PQ, CanadaAsia Biega - Microsoft Research (India)Michael Ekstrand - Boise State UniversityBen Carterette - Photon Spot (United States)Emine Yilmaz - UCL, London, EnglandACM
- Publication Details
- PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE 2021 (WWW 2021), pp 1065-1075
- Publisher
- Assoc Computing Machinery
- Number of pages
- 11
- Grant note
- EP/P024289/1 / EPSRC; UK Research & Innovation (UKRI); Engineering & Physical Sciences Research Council (EPSRC) IIS 17-51278 / National Science Foundation; National Science Foundation (NSF)
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Information Science
- Web of Science ID
- WOS:000733621801007
- Scopus ID
- 2-s2.0-85107929926
- Other Identifier
- 991021818387904721
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Industry collaboration
- Domestic collaboration
- International collaboration
- Web of Science research areas
- Computer Science, Artificial Intelligence
- Computer Science, Information Systems
- Computer Science, Interdisciplinary Applications
- Computer Science, Theory & Methods