Preprint
Exploring the Curious Case of Code Prompts
arXiv
25 Apr 2023
Abstract
Recent work has shown that prompting language models with code-like
representations of natural language leads to performance improvements on
structured reasoning tasks. However, such tasks comprise only a small subset of
all natural language tasks. In our work, we seek to answer whether or not
code-prompting is the preferred way of interacting with language models in
general. We compare code and text prompts across three popular GPT models
(davinci, code-davinci-002, and text-davinci-002) on a broader selection of
tasks (e.g., QA, sentiment, summarization) and find that with few exceptions,
code prompts do not consistently outperform text prompts. Furthermore, we show
that the style of code prompt has a large effect on performance for some but
not all tasks and that fine-tuning on text instructions leads to better
relative performance of code prompts.
Metrics
8 Record Views
Details
- Title
- Exploring the Curious Case of Code Prompts
- Creators
- Li Zhang - Drexel University, Computer ScienceLiam DuganHainiu Xu - University of PennsylvaniaChris Callison-Burch - University of Pennsylvania
- Publication Details
- arXiv
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Computer Science
- Other Identifier
- 991022122861604721