How to evaluate predictive AI for deterioration and sepsis
Predictive tools watch vital signs, labs and notes and raise an alert before a patient worsens. Their value depends less on headline accuracy than on three things: how many alerts they create, how early they fire compared with your team, and what happens after an alert.
The same questions apply whether the model is built into your EHR or bought from a third party.
Last reviewed November 2026 · Independent and vendor-neutral
What does the tool you’re evaluating do?
Jump to the matching checklist. Some answers lead to a different guide.
Physicians and nurses usually want
- Real lead time over what the team would notice anyway
- Few false alarms
- A reason the score is high
- A clear action for each alert
Administrators usually want
- Change in mortality, ICU transfers and length of stay
- Sepsis bundle compliance
- Nursing workload
- Clarity on liability when an alert is not acted on
Questions for every predictive tool
Applies if deterioration, sepsis and condition-specific models.
Questions to ask
- At the threshold you recommend, what are the positive predictive value and the number of alerts per 100 patient-days?
- How was the outcome defined: by clinical criteria or by billing codes? Billing-code outcomes can inflate apparent accuracy.
- Is the model calibrated, not just good at ranking (AUC)? Ask for a calibration plot.
- Was it validated externally, at hospitals other than the developer's, and on outcomes measured after the alert time?
- If the model is supplied in a certified EHR, ask for its ONC HTI-1 source attributes.
- How do we retune the threshold, and who monitors drift?
Test it locally
- Run the model silently for 2 to 3 months before anyone sees alerts.
- At several thresholds, measure alerts per day per unit, PPV, sensitivity and median lead time.
- Choose the threshold together with nursing and physician leaders, and write down the response expected for each alert.
Red flags
- Outcomes defined by billing codes
- No calibration data
- Alerts with no recommended action
- No estimate of alert volume per unit
General deterioration warnings
Applies if the tool gives a general early-warning score for ward or ICU patients.
Questions to ask
- Does it beat the score you already use (for example a standard early-warning score) on your data?
- Who receives the alert: bedside nurse, rapid-response team or a remote monitoring center?
- How is alert burden balanced across shifts and units?
Sepsis prediction
Applies if the tool predicts or helps diagnose sepsis.
Questions to ask
- Is it FDA-authorized? If so, read the labeling: authorized sepsis tools are cleared as aids, not as the sole basis for diagnosis.
- Which sepsis definition was used for training and testing?
- How many alerts fire after treatment has already started? Late alerts add workload without benefit.
Single-condition predictions
Applies if the tool predicts one specific event, such as acute kidney injury, falls or readmission.
Questions to ask
- Is there an action that changes the outcome once a patient is flagged, and do you have the staff to take it?
- How does it perform in subgroups that matter for that condition (for example baseline kidney disease)?
What published studies show
A 2021 external validation of a widely deployed proprietary sepsis model found a hospital-level AUC of 0.63, compared with 0.76 to 0.83 reported by the developer. At the threshold studied, it alerted on 18% of hospitalizations and missed 67% of sepsis cases. The developer disputed the configuration used. The lesson for buyers is local validation and threshold tuning, not any single product.
JAMA Internal Medicine, 2021Further reading
Browse hospital medicine & critical care tools
Every listing shows regulatory status, published evidence and named health-system users where public.