Logo image
Back
Choice-75: A Dataset on Decision Branching in Script Learning
Conference proceeding   Open access

Choice-75: A Dataset on Decision Branching in Script Learning

Zhaoyi Joey Hou, Li Zhang and Chris Callison-Burch
PROCEEDINGS OF THE 2024 JOINT INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS, LANGUAGE RESOURCES AND EVALUATION, LREC-COLING 2024, pp 3215-3223
01 Jan 2024
url
https://aclanthology.org/2024.lrec-main.285/View
Published, Version of Record (VoR)CC BY V4.0,  Open

Abstract

Computer Science, Artificial Intelligence Computer Science, Interdisciplinary Applications Language & Linguistics Linguistics Science & Technology Computer Science Social Sciences Technology
Script learning studies how stereotypical events unfold, enabling machines to reason about narratives with implicit information. Previous works mostly consider a script as a linear sequence of events while ignoring the potential branches that arise due to people's circumstantial choices. We hence propose Choice-75, the first benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios. We also present preliminary results with current large language models (LLM). Although they demonstrate overall decent performance, there is still notable headroom in hard scenarios.

Metrics

1 Record Views
1 citations in Web of Science

Details

Logo image