Logo image
Back
Causal Reasoning About Entities and Events in Procedural Texts
Conference proceeding   Open access

Causal Reasoning About Entities and Events in Procedural Texts

Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora and Chris Callison-Burch
Findings of the Association for Computational Linguistics: EACL 2023, pp 415-431
01 Jan 2023
DOI: https://doi.org/10.18653/v1/2023.findings-eacl.31
url
https://doi.org/10.18653/v1/2023.findings-eacl.31View
Published, Version of Record (VoR) Open CC BY V4.0

Abstract

Computer Science, Artificial Intelligence Computer Science, Theory & Methods Science & Technology Computer Science Technology
Entities and events are crucial to natural language reasoning and common in procedural texts. Existing work has focused either exclusively on entity state tracking (e.g., whether a pan is hot) or on event reasoning (e.g., whether one would burn themselves by touching the pan), while these two tasks are often causally related. We propose CREPE, the first benchmark on causal reasoning of event plausibility and entity states. We show that most language models, including GPT-3, perform close to chance at .35 F1, lagging far behind human at .87 F1. We boost model performance to .59 F1 by creatively representing events as programming languages while prompting language models pretrained on code. By injecting the causal relations between entities and events as intermediate reasoning steps in our representation, we further boost the performance to .67 F1. Our findings indicate not only the challenge that CREPE brings for language models, but also the efficacy of code-like prompting combined with chain-of-thought prompting for multihop event reasoning.(1)

Metrics

5 Record Views

Details

InCites Highlights

Data related to this publication, from InCites Benchmarking & Analytics tool:

Collaboration types
Domestic collaboration
Web of Science research areas
Computer Science, Artificial Intelligence
Computer Science, Theory & Methods
Logo image