Conference proceeding
Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs
CONFERENCE ON ROBOT LEARNING, VOL 229, v 229
01 Jan 2023
Abstract
We present an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework for grounding a variety of entities, such as object instances, agents, and regions, with free-form text-based queries. Unlike conventional semantic-based object localization approaches, our system facilitates context-aware entity localization, allowing for queries such as "pick up a cup on a kitchen table" or "navigate to a sofa on which someone is sitting". In contrast to existing research on 3D scene graphs, OVSG supports free-form text input and open-vocabulary querying. Through a series of comparative experiments using the ScanNet [1] dataset and a self-collected dataset, we demonstrate that our proposed approach significantly surpasses the performance of previous semantic-based localization techniques. Moreover, we highlight the practical application of OVSG in real-world robot navigation and manipulation experiments. The code and dataset used for evaluation can be found at https://github.com/changhaonan/OVSG.
Metrics
1 Record Views
Details
- Title
- Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs
- Creators
- Haonan Chang - University of New BrunswickKowndinya Boyalakuntla - University of New BrunswickShiyang Lu - University of New BrunswickSiwei Cai - Drexel UniversityEric Pu Jing - University of New BrunswickShreesh Keskar - University of New BrunswickShijie Geng - University of New BrunswickAdeeb Abbas - Drexel UniversityLifeng Zhou - Drexel UniversityKostas Bekris - University of New BrunswickAbdeslam Boularias - University of New Brunswick
- Contributors
- J Tan (Editor)M Toussaint (Editor)K Darvish (Editor)
- Publication Details
- CONFERENCE ON ROBOT LEARNING, VOL 229, v 229
- Series
- Proceedings of Machine Learning Research
- Publisher
- JMLR-JOURNAL MACHINE LEARNING RESEARCH
- Number of pages
- 25
- Grant note
- 1846043; 2132972 / NSF
- Resource Type
- Conference proceeding
- Language
- English
- Academic Unit
- Electrical and Computer Engineering
- Web of Science ID
- WOS:001221201502002
- Scopus ID
- 2-s2.0-85174051892
- Other Identifier
- 991022207073704721