Preprint
PROC2PDDL: Open-Domain Planning Representations from Texts
ArXiv.org
29 Feb 2024
Abstract
Planning in a text-based environment continues to be a major challenge for AI
systems. Recent approaches have used language models to predict a planning
domain definition (e.g., PDDL) but have only been evaluated in closed-domain
simulated environments. To address this, we present Proc2PDDL , the first
dataset containing open-domain procedural texts paired with expert-annotated
PDDL representations. Using this dataset, we evaluate state-of-the-art models
on defining the preconditions and effects of actions. We show that Proc2PDDL is
highly challenging, with GPT-3.5's success rate close to 0% and GPT-4's around
35%. Our analysis shows both syntactic and semantic errors, indicating LMs'
deficiency in both generating domain-specific prgorams and reasoning about
events. We hope this analysis and dataset helps future progress towards
integrating the best of LMs and formal planning.
Metrics
13 Record Views
Details
- Title
- PROC2PDDL: Open-Domain Planning Representations from Texts
- Creators
- Tianyi Zhang - University of PennsylvaniaLi Zhang - Drexel University, Computer ScienceZhaoyi HouZiyu WangYuling GuPeter ClarkChris Callison-Burch - University of PennsylvaniaNiket Tandon - Allen Institute
- Publication Details
- ArXiv.org
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Computer Science
- Other Identifier
- 991022122861904721