Conference paper
Hierarchical LLMs in-the-Loop Optimization for Real-Time Multi-Robot Target Tracking Under Unknown Hazards
2025 IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS), pp 1-7
04 Dec 2025
Abstract
Real-time multi-robot coordination in hazardous and adversarial environments requires fast, reliable adaptation to dynamic threats. While Large Language Models (LLMs) offer strong high-level reasoning capabilities, the lack of safety guarantees limits their direct use in critical decision-making. In this paper, we propose a hierarchical optimization framework that integrates LLMs into the decision loop for multi-robot target tracking in dynamic and hazardous environments. Rather than generating control actions directly, LLMs are used to generate task configuration and adjust parameters in a bi-level task allocation and planning problem. We formulate multi-robot coordination for tracking tasks as a bi-level optimization problem, with LLMs to reason about potential hazards in the environment and the status of the robot team and modify both the inner and outer levels of the optimization. This hierarchical approach enables real-time adjustments to the robots' behavior. Additionally, a human supervisor can offer broad guidance and assessments to address unexpected dangers, model mismatches, and performance issues arising from local minima. We validate our proposed framework in both simulation and real-world experiments with comprehensive evaluations, demonstrating its effectiveness and showcasing its capability for safe LLM integration with multi-robot systems.
Metrics
1 Record Views
Details
- Title
- Hierarchical LLMs in-the-Loop Optimization for Real-Time Multi-Robot Target Tracking Under Unknown Hazards
- Creators
- Yuwei Wu - University of PennsylvaniaYuezhan Tao - University of PennsylvaniaPeihan Li - Drexel University, Electrical and Computer EngineeringGuangyao Shi - University of Southern CaliforniaGaurav S. Sukhatme - University of Southern CaliforniaVijay Kumar - University of PennsylvaniaLifeng Zhou - Drexel University, Electrical and Computer Engineering
- Publication Details
- 2025 IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS), pp 1-7
- Conference
- 2025 IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS) (Singapore, Singapore, 04 Dec 2025–05 Dec 2025)
- Publisher
- IEEE
- Number of pages
- 7
- Grant note
- W911NF-17-2-0181 / ARL (10.13039/100005423) CCR-2112665 / NSF (10.13039/100000001)
- Resource Type
- Conference paper
- Language
- English
- Academic Unit
- Electrical and Computer Engineering
- Web of Science ID
- WOS:001706372600022
- Scopus ID
- 2-s2.0-105032928367
- Other Identifier
- 9798331593599; 991022170440504721