Journal article
Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in Dynamic Clinical Settings
Proceedings of ACM on interactive, mobile, wearable and ubiquitous technologies, v 7(1), 12
28 Mar 2023
Featured in Collection : UN Sustainable Development Goals @ Drexel
Abstract
In clinical settings, most automatic recognition systems use visual or sensory data to recognize activities. These systems cannot recognize activities that rely on verbal assessment, lack visual cues, or do not use medical devices. We examined speech-based activity and activity-stage recognition in a clinical domain, making the following contributions. (1) We collected a high-quality dataset representing common activities and activity stages during actual trauma resuscitation events-the initial evaluation and treatment of critically injured patients. (2) We introduced a novel multimodal network based on audio signal and a set of keywords that does not require a high-performing automatic speech recognition (ASR) engine. (3) We designed novel contextual modules to capture dynamic dependencies in team conversations about activities and stages during a complex workflow. (4) We introduced a data augmentation method, which simulates team communication by combining selected utterances and their audio clips, and showed that this method contributed to performance improvement in our data-limited scenario. In offline experiments, our proposed context-aware multimodal model achieved F-1-scores of 73.2 +/- 0.8% and 78.1 +/- 1.1% for activity and activity-stage recognition, respectively. In online experiments, the performance declined about 10% for both recognition types when using utterance-level segmentation of the ASR output. The performance declined about 15% when we omitted the utterance-level segmentation. Our experiments showed the feasibility of speech-based activity and activity-stage recognition during dynamic clinical events.
Metrics
Details
- Title
- Real-time Context-Aware Multimodal Network for Activity and Activity-Stage Recognition from Team Communication in Dynamic Clinical Settings
- Creators
- Chenyang Gao - Rutgers, The State University of New JerseyIvan Marsic - Rutgers, The State University of New JerseyAleksandra Sarcevic - Drexel UniversityWaverly Gestrich-Thompson - Children's NationalRandall S. Burd - Children's National
- Publication Details
- Proceedings of ACM on interactive, mobile, wearable and ubiquitous technologies, v 7(1), 12
- Publisher
- Assoc Computing Machinery
- Number of pages
- 28
- Grant note
- King Saud University EP/S035362/1; EP/R033439/1 / EPSRC; UK Research & Innovation (UKRI); Engineering & Physical Sciences Research Council (EPSRC)
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Information Science
- Web of Science ID
- WOS:000957429700012
- Scopus ID
- 2-s2.0-85152487441
- Other Identifier
- 991020476622404721
UN Sustainable Development Goals (SDGs)
This publication has contributed to the advancement of the following goals:
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- Web of Science research areas
- Computer Science, Information Systems
- Engineering, Electrical & Electronic
- Telecommunications