Logo image
Back
TextAidAR: Text-driven domain knowledge for fine-grained medical activity recognition
Journal article   Peer reviewed

TextAidAR: Text-driven domain knowledge for fine-grained medical activity recognition

Wenjin Zhang, Chenyang Gao, Sifan Yuan, Mary S. Kim, Aleksandra Sarcevic, Randall S. Burd and Ivan Marsic
Computer vision and image understanding, v 272, 104926
Aug 2026
DOI: https://doi.org/10.1016/j.cviu.2026.104926

Abstract

Activity recognition Few-shot learning Medical activity recognition
While recent advances in activity recognition have achieved remarkable success, these methods depend on large-scale datasets with extensive annotations. Developing fine-grained activity recognition for domain-specific tasks presents unique challenges, including limited labeled data due to privacy concerns and labor-intensive annotation processes. Distinguishing visually similar activities remains difficult, as fine-grained actions often involve subtle differences (i.e., slight variations in hand positioning and tiny object interactions). In trauma resuscitation, critical procedures such as intravenous (IV) placement and temperature measurement occur within small regions of the scene and in complex, crowded environments, further complicating activity recognition. To address these challenges, we reframed medical activity recognition as an image classification task instead of relying on temporal modeling. We introduce TextAidAR, an activity recognition method that incorporates expert-defined domain knowledge to improve visual representation learning. By leveraging vision-text alignment, our approach enhances spatial feature learning using structured medical descriptions provided by domain experts. We also introduce a lightweight module, the MedSpec adapter, to refine pre-trained text embeddings for medical contexts and propose two key loss functions—class-conditioned alignment loss and MedDistinct loss—to enhance the robustness of visual-text alignment. Our studies show that TextAidAR surpasses existing video-based and image-based methods, achieving state-of-the-art performance with a mAP of 0.72 in detecting 13 fine-grained medical activities in real-world trauma resuscitation. On the NurViD dataset, TextAidAR achieves a 41.2% improvement in overall performance compared to the strongest existing baseline. [Display omitted] •We developed TextAidAR, an approach to fine-grained medical activity recognition by leveraging medical domain knowledge through vision-text alignment.•We reframed medical activity recognition as an image classification task, eliminating the need for complex temporal modeling while achieving robust performance.•We incorporated expert-written medical descriptions of medical activities to enrich visual feature learning, enhancing the model’s ability to recognize fine-grained activities.•We introduced a lightweight MedSpec adapter to refine pre-trained text embeddings for medical contexts, alongside class-conditioned alignment loss and MedDistinct loss to improve robustness.•We conducted experiments on a real-world trauma resuscitation dataset and NurViD dataset, showing that our method outperforms video-based models, image-based methods, and prior activity recognition approaches.

Metrics

1 Record Views

Details

Logo image