Conference proceeding
GUIDELLM: Exploring LLM-Guided Conversation with Applications in Autobiography Interviewing
Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies: Long Papers, NAACL-HLT 2025, v 1, pp 5558-5588
2025
Abstract
Although Large Language Models (LLMs) succeed in human-guided conversations such as instruction following and question answering, the potential of LLM-guided conversations—where LLMs direct the discourse and steer the conversation’s objectives—remains under-explored. In this study, we first characterize LLM-guided conversation into three fundamental components: (i) Goal Navigation; (ii) Context Management; (iii) Empathetic Engagement, and propose GuideLLM as an installation. We then implement an interviewing environment for the evaluation of LLM-guided conversation. Specifically, various topics are involved in this environment for comprehensive interviewing evaluation, resulting in around 1.4k turns of utterances, 184k tokens, and over 200 events mentioned during the interviewing for each chatbot evaluation. We compare GuideLLM with 6 state-of-the-art LLMs such as GPT-4o and Llama-3-70b-Instruct, from the perspective of interviewing quality, and autobiography generation quality. For automatic evaluation, we derive user proxies from multiple autobiographies and employ LLM-as-a-judge to score LLM behaviors. We further conduct a human-involved experiment by employing 45 human participants to chat with GuideLLM and baselines. We then collect human feedback, preferences, and ratings regarding the qualities of conversation and autobiography. Experimental results indicate that GuideLLM significantly outperforms baseline LLMs in automatic evaluation and achieves consistent leading performances in human ratings.
Metrics
1 Record Views
Details
- Title
- GUIDELLM: Exploring LLM-Guided Conversation with Applications in Autobiography Interviewing
- Creators
- Jinhao Duan - Drexel University, United StatesXinyu Zhao - University of North Carolina at Chapel HillZhuoxuan Zhang - Brown UniversityEunhye Ko - The University of Texas at AustinLily Boddy - The University of Texas at AustinChenan Wang - Drexel UniversityTianhao Li - The University of Texas at AustinAlexander Rasgon - The University of Texas at AustinJunyuan Hong - The University of Texas at AustinMin Kyung Lee - The University of Texas at AustinChenxi Yuan - New Jersey Institute of TechnologyQi Long - University of PennsylvaniaYing Ding - The University of Texas at AustinTianlong Chen - University of North Carolina at Chapel HillKaidi Xu - Drexel University, United States
- Publication Details
- Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies: Long Papers, NAACL-HLT 2025, v 1, pp 5558-5588
- Conference
- 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Albuquerque, New Mexico, United States, 29 Apr 2025–04 May 2025)
- Publisher
- Association for Computational Linguistics
- Number of pages
- 31
- Grant note
- 2319242 / National Sleep Foundation (100003187) National Science Foundation (http://data.elsevier.com/vocabulary/SciValFunders/100000001) OT2OD032581 / National Institutes of Health (100000002) OpenAI (http://data.elsevier.com/vocabulary/SciValFunders/100025178) OTA-21-008 / National Institutes of Health (100000002) R01LM014306-01 / National Institutes of Health (100000002) OT2OD032581; OTA-21-008; R01LM014306-01 / NIH 2319242 / National Science Foundation (http://data.elsevier.com/vocabulary/SciValFunders/100000001)
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Computer Science
- Scopus ID
- 2-s2.0-105027443807
- Other Identifier
- 991022197433404721