Logo image
Back
Small but Mighty: New Benchmarks for Split and Rephrase
Conference proceeding   Open access

Small but Mighty: New Benchmarks for Split and Rephrase

Li Zhang, Huaiyu Zhu, Siddhartha Brahma and Yunyao Li
PROCEEDINGS OF THE 2020 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (EMNLP), pp 1198-1205
01 Jan 2020
DOI: https://doi.org/10.18653/v1/2020.emnlp-main.91
url
https://doi.org/10.18653/v1/2020.emnlp-main.91View
Published, Version of Record (VoR) Open

Abstract

Computer Science, Artificial Intelligence Science & Technology Computer Science Technology
Split and Rephrase is a text simplification task of rewriting a complex sentence into simpler ones. As a relatively new task, it is paramount to ensure the soundness of its evaluation benchmark and metric. We find that the widely used benchmark dataset universally contains easily exploitable syntactic cues caused by its automatic generation process. Taking advantage of such cues, we show that even a simple rule-based model can perform on par with the state-of-the-art model. To remedy such limitations, we collect and release two crowdsourced benchmark datasets. We not only make sure that they contain significantly more diverse syntax, but also carefully control for their quality according to a well-defined set of criteria. While no satisfactory automatic metric exists, we apply fine-grained manual evaluation based on these criteria using crowdsourcing, showing that our datasets better represent the task and are significantly more challenging for the models.(1)

Metrics

3 Record Views
3 citations in Web of Science

Details

InCites Highlights

Data related to this publication, from InCites Benchmarking & Analytics tool:

Collaboration types
Industry collaboration
Domestic collaboration
Web of Science research areas
Computer Science, Artificial Intelligence
Logo image