Logo image
Back
Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domains
Book chapter

Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domains

Manil Shrestha and Edward Kim
Artificial Intelligence in Medicine, pp 367-371
08 Jul 2026
DOI: https://doi.org/10.1007/978-3-032-30813-9_68
Featured in Collection :   Drexel's Newest Publications

Abstract

Conformal Prediction Entity Extraction Large Language Models
Large Language Models (LLMs) are increasingly used for medical entity extraction, yet their confidence scores are often miscalibrated, limiting safe deployment in clinical settings. We present a conformal prediction framework that provides finite-sample coverage guarantees for LLM-based extraction across two clinical domains. First, we extract structured entities from 1,000 FDA drug labels across eight sections using GPT-4.1, verified via FactScore-based atomic statement evaluation (97.7% accuracy over 110,664 entities). Second, we extract radiological entities from MIMIC-CXR reports using the RadGraph schema with GPT-4.1 and Llama-4-Maverick, evaluated against physician annotations (entity F1: 0.81 to 0.84). Our central finding is that miscalibration direction reverses across domains: on well-structured FDA labels, models are underconfident, requiring modest conformal thresholds, while on free-text radiology reports, models are overconfident, demanding strict thresholds (ττ̂ approaching 1.0). Despite this heterogeneity, conformal prediction achieves coverage close to the (1-α) (1-α ) target on held-out test sets in both settings with manageable rejection rates (9–13%). These results demonstrate that calibration is not a global model property but varies across document types, extraction categories, and model architectures, motivating domain-specific conformal calibration for safe clinical deployment.

Metrics

1 Record Views

Details

Logo image