Improving sepsis best practice utility and clinical acceptance using an LLM-enhanced prediction system
TL;DR
CompOSER-LLM, an enhanced sepsis prediction system that integrates a large language model (LLM) that analyzes clinical notes to provide contextual support for sepsis risk assessment in two emergency departments within a large academic health system, is implemented.
Abstract
Early detection of sepsis is essential for reducing mortality, but diagnosis can be delayed because symptoms of sepsis are often non-specific, frequently mimic other conditions and differ between individuals. We implemented COMPOSER-LLM, an enhanced sepsis prediction system that integrates a large language model (LLM) that analyzes clinical notes to provide contextual support for sepsis risk assessment in two emergency departments within a large academic health system. Using Bayesian causal impact analysis, we evaluated how LLM augmentation influenced nurses’ clinical perception of best practice advisory (BPA) relevance, using the rate of BPAs acknowledged as “No Infection Suspected” as a proxy for BPA clinical acceptance. Additionally, a 9 question survey instrument was sent to nurses immediately after the COMPOSER-LLM BPA was fired, in order to assess the perceived utility and impact of the BPA on nursing care. Post-deployment, the rate of “No Infection Suspected” acknowledgements was significantly lower than would be expected based on pre-deployment trends. Survey respondents reported that the COMPOSER-LLM BPA was useful, helped identify at-risk patients who might otherwise be missed, and was generally trusted by nursing staff. Nurses with 0–5 years of experience were significantly more likely than more experienced nurses to believe patients had infection after an BPA and to feel the BPA increased their expectation that a patient would develop sepsis, while no other significant experience-based differences were observed.