Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Preprint
GuideLLM: Exploring LLM-Guided Conversation with Applications in Autobiography Interviewing
IACAPAP ArXiv (Online)
10 Feb 2025
Abstract
Although Large Language Models (LLMs) succeed in human-guided conversations
such as instruction following and question answering, the potential of
LLM-guided conversations-where LLMs direct the discourse and steer the
conversation's objectives-remains under-explored. In this study, we first
characterize LLM-guided conversation into three fundamental components: (i)
Goal Navigation; (ii) Context Management; (iii) Empathetic Engagement, and
propose GuideLLM as an installation. We then implement an interviewing
environment for the evaluation of LLM-guided conversation. Specifically,
various topics are involved in this environment for comprehensive interviewing
evaluation, resulting in around 1.4k turns of utterances, 184k tokens, and over
200 events mentioned during the interviewing for each chatbot evaluation. We
compare GuideLLM with 6 state-of-the-art LLMs such as GPT-4o and
Llama-3-70b-Instruct, from the perspective of interviewing quality, and
autobiography generation quality. For automatic evaluation, we derive user
proxies from multiple autobiographies and employ LLM-as-a-judge to score LLM
behaviors. We further conduct a human-involved experiment by employing 45 human
participants to chat with GuideLLM and baselines. We then collect human
feedback, preferences, and ratings regarding the qualities of conversation and
autobiography. Experimental results indicate that GuideLLM significantly
outperforms baseline LLMs in automatic evaluation and achieves consistent
leading performances in human ratings.
Metrics
31 Record Views
Details
- Title
- GuideLLM: Exploring LLM-Guided Conversation with Applications in Autobiography Interviewing
- Creators
- Jinhao DuanXinyu ZhaoZhuoxuan ZhangEunhye KoLily BoddyChenan WangTianhao LiAlexander RasgonJunyuan HongMin Kyung LeeChenxi YuanQi LongYing DingTianlong ChenKaidi Xu
- Publication Details
- IACAPAP ArXiv (Online)
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Computer Science (Computing); Communication
- Other Identifier
- 991022027537104721