Journal article
Rethinking the evaluation of news algorithms: aligning epistemic standards, user priorities and evaluation metrics in recommender system design
Frontiers in communication, v 11, 1844993
01 Jun 2026
Abstract
As media organizations increasingly deploy recommender systems, these technologies play a growing role in shaping how individuals encounter and engage with news and public information online. Designing and evaluating these systems effectively, particularly in the news domain, requires attention to the challenges users face in forming knowledge and justified beliefs within complex digital information environments. To address this need, we draw on the normative concept of epistemic welfare, which offers a systematic, veritistically grounded approach to assessing how recommender systems support users’ capacity to access, interpret, and use information. Using large-scale online surveys in Belgium and Switzerland (N = 3,076), this study investigates how users prioritize distinct epistemic standards in algorithmic recommendations of news vs. entertainment from public service vs. private media across two structurally similar contexts, with entertainment recommendations serving as a comparative reference category. Results show a consistent prioritization of reliability across all contexts, including entertainment, while efficiency and speed are systematically deprioritized. We integrate theoretical and empirical insights into the Epistemic Welfare Evaluation Framework for News Recommender Systems (EWEF-NR), an exploratory conceptual framework that connects normative epistemic standards and empirical user priorities with established evaluation metrics. Bridging social scientific and computational perspectives, this paper provides an interdisciplinary user-centered approach for designing and assessing news recommender systems that strengthen users’ epistemic agency.
Metrics
1 Record Views
Details
- Title
- Rethinking the evaluation of news algorithms: aligning epistemic standards, user priorities and evaluation metrics in recommender system design
- Creators
- Michelle Kulig (Corresponding Author) - University of FribourgWillem Buyens - University of AntwerpRaphaël Tuor - University of FribourgCélina Treuillier - University of FribourgManuel Puppis - University of FribourgHilde Van den Bulck - Drexel UniversitySteve Paulussen - University of AntwerpDenis Lalanne - University of Fribourg
- Publication Details
- Frontiers in communication, v 11, 1844993
- Publisher
- Frontiers Media S.A
- Number of pages
- 11
- Grant note
- Schweizerischer Nationalfonds zur Frderung der Wissenschaftlichen Forschung: 100019E_212521 Fonds Wetenschappelijk Onderzoek: G078625N
The author(s) declared that financial support was received for this work and/or its publication. The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was funded by the Fonds Wetenschappelijk Onderzoek Vlaanderen (FWO project G078625N) and the Swiss National Science Foundation (SNSF project 100019E_212521). The funders had no role in the design of the study, data collection, analysis, interpretation, or the writing of this article.
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Communication
- Web of Science ID
- WOS:001810825200001
- Other Identifier
- 991022192896304721