Files and links (1)
Published, Version of Record (VoR)CC BY V4.0, Open
Conference proceeding
OPENPI2.0: An Improved Dataset for Entity Tracking in Texts
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp 166-178
01 Jan 2024
Abstract
Much texts describe a changing world (e.g., procedures, stories, newswires), and understanding them requires tracking how entities change. An earlier dataset, OpenPI, provided crowdsourced annotations of entity state changes in text. However, a major limitation was that those annotations were free-form and did not identify salient changes, hampering model evaluation. To overcome these limitations, we present an improved dataset, OpenPI2.0, where entities and attributes are fully canonicalized and additional entity salience annotations are added. On our fairer evaluation setting, we find that current state-of-the-art language models are far from competent. We also show that using state changes of salient entities as a chain-of-thought prompt, downstream performance is improved on tasks such as question answering and classical planning, outperforming the setting involving all related entities indiscriminately. We offer OpenPI2.0 for the continued development of models that can understand the dynamics of entities in text.
Metrics
5 Record Views
Details
- Title
- OPENPI2.0: An Improved Dataset for Entity Tracking in Texts
- Creators
- Li Zhang - University of PennsylvaniaHainiu Xu - University of PennsylvaniaAbhinav Kommula - University of California, BerkeleyChris Callison-Burch - University of PennsylvaniaNiket Tandon - Allen Institute
- Contributors
- Y Graham (Editor)M Purver (Editor)
- Publication Details
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp 166-178
- Conference
- Conference of the European Chapter of the Association for Computational Linguistics, 18 (St. Julians, Malta, 17 Mar 2024–22 Mar 2024)
- Publisher
- Association for Computational Linguistics
- Number of pages
- 13
- Grant note
- 2022-22072200005 / Office of the Director of National Intelligence (ODNI) via the IARPA HIATUS Program FA8750-23-C-0507 / AFRL; United States Department of Defense; US Air Force Research Laboratory 1928631 / NSF; National Science Foundation (NSF) FA8750-19-2-1004 / DARPA KAIROS Program; United States Department of Defense
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Computer Science
- Web of Science ID
- WOS:001356732600010
- Other Identifier
- 991022123353404721
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- Web of Science research areas
- Computer Science, Artificial Intelligence
- Computer Science, Interdisciplinary Applications
- Computer Science, Theory & Methods