Conference proceeding
Opportunities or Risks to Reduce Labor in Crowdsourcing Translation? Characterizing Cost Versus Quality via a PageRank-HITS Hybrid Model
PROCEEDINGS OF THE TWENTY-FOURTH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE (IJCAI), pp.1025-1032
01 Jan 2015
Abstract
Crowdsourcing machine translation shows advantages of lower expense in money to collect the translated data. Yet, when compared with translation by trained professionals, results collected from non-professional translators might yield low-quality outputs. A general solution for crowdsourcing practitioners is to employ a large amount of labor force to gather enough redundant data and then solicit from it. Actually we can further save money by avoid collecting bad translations. We propose to score Turkers by their authorities during observation, and then stop hiring the unqualified Turkers. In this way, we bring both opportunities and risks in crowdsourced translation: we can make it cheaper than cheaper while we might suffer from quality loss. In this paper, we propose a graph-based PageRank-HITS Hybrid model to distinguish authoritative workers from unreliable ones. The algorithm captures the intuition that good translation and good workers are mutually reinforced iteratively in the proposed frame. We demonstrate the algorithm will keep the performance while reduce work force and hence cut cost. We run experiments on the NIST 2009 Urdu-to-English evaluation set with Mechanical Turk, and quantitatively evaluate the performance in terms of BLEU score, Pearson correlation and real money.
Metrics
8 Record Views
Details
- Title
- Opportunities or Risks to Reduce Labor in Crowdsourcing Translation? Characterizing Cost Versus Quality via a PageRank-HITS Hybrid Model
- Creators
- Rui Yan - Univ Penn, Dept Comp & Informat Sci, Philadelphia, PA 19104 USAYiping Song - Peking Univ, Dept Comp Sci, Beijing, Peoples R ChinaCheng-Te Li - Acad Sinica, Taipei, TaiwanMing Zhang - Peking Univ, Dept Comp Sci, Beijing, Peoples R ChinaXiaohua Hu - Drexel Univ, Coll Comp & Informat, Philadelphia, PA 19104 USA
- Contributors
- Q Yang (Editor)M Wooldridge (Editor)
- Publication Details
- PROCEEDINGS OF THE TWENTY-FOURTH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE (IJCAI), pp.1025-1032
- Conference
- TWENTY-FOURTH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE (IJCAI), 24th
- Publisher
- Ijcai-Int Joint Conf Artif Intell
- Number of pages
- 8
- Grant note
- 61472006 / National Natural Science Foundation of China (NSFC) 2014CB340405 / National Basic Research Program (973 Program); National Basic Research Program of China 20130001110032 / Doctoral Program of Higher Education of China; Specialized Research Fund for the Doctoral Program of Higher Education (SRFDP)
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Information Science (Informatics)
- Identifiers
- 991019170325104721
InCites Highlights
These are selected metrics from InCites Benchmarking & Analytics tool, related to this output
- Collaboration types
- Domestic collaboration
- International collaboration
- Web of Science research areas
- Computer Science, Artificial Intelligence
- Computer Science, Interdisciplinary Applications