CIFoundry

How to evaluate a medical AI tool

Most AI tools look good in a demo. Whether one works in your hospital depends on questions the demo rarely answers: what exactly it was cleared or built to do, what evidence stands behind it, how it performs on your patients and equipment, and who watches it after go-live.

This guide covers the questions that apply to every tool. Because "medical AI" covers very different products, the specialty guides then split the questions by what the tool actually does. An ambient scribe used by cardiologists is judged like a scribe, not like an echo algorithm.

Last reviewed November 2026 · Independent and vendor-neutral

Find the guide for your tool

Pick the area, then check the line underneath matches what the tool does. Inside, each guide splits its checklist by tool type.

Before you look at any product

Start with the problem

  • Write down the problem and a measurable target first, for example "cut after-hours charting by 30 minutes a day" or "shorten time to treatment for large-vessel stroke". A product looking for a use case is a common reason pilots stall.
  • Decide in advance what result would count as success, and what result would make you stop.

Set up ownership

  • Use a multidisciplinary group: clinicians, nursing, informatics, IT security, compliance, legal and quality. Many systems fold AI into existing quality and safety committees rather than creating a new one.
  • Bring frontline users in at the start, not at go-live. Clinicians who help choose a tool are more likely to use it and spot workflow problems early.
  • Keep one inventory of every AI tool in use: owner, intended use, version, validation results and monitoring schedule. Report AI problems through your existing safety-event system.

What FDA clearance does and does not tell you

Tools that diagnose, detect or triage disease are usually regulated as software medical devices. Administrative tools, and most ambient scribes, generally are not. Either way, read what the status actually covers.

Check

  • The exact intended use in the FDA summary: which patients, which inputs (scanner, ECG device, image type), which user, and whether the tool assists a clinician or acts on its own. Using it outside that indication is the most common hidden risk.
  • The pathway. About 97% of AI devices are cleared through 510(k), which rests on similarity to an earlier device and often needs no new prospective clinical data. Clearance means it can be sold for that use, not that it has been shown to improve care.
  • Whether the tool has an FDA-authorized Predetermined Change Control Plan (PCCP), which allows certain model updates after clearance, and how you will be told when the model changes.
  • For a predictive tool supplied inside a certified EHR, ask for the ONC HTI-1 "source attributes" (intended use, training data, fairness approach, validation, update schedule). They make a good question list for any predictive tool.
  • If you have sites in the EU, AI used for diagnosis is high-risk under the EU AI Act, with added duties on oversight and documentation.

More on 510(k), De Novo and PMA, and how to check a record yourself, in our explainer. Read that guide →

The 12 questions that apply to every tool

Fit and evidence

  • Intended use. Does the stated use match the patients, setting, user and decision you plan to use it for?
  • Evidence level. From weakest to strongest: vendor-internal test, retrospective external validation, prospective "silent" run, prospective live study, randomized trial, proven change in outcomes. Ask which level the claims rest on and look for peer-reviewed external validation.
  • Population and equipment. Ask for results by age, sex, race and ethnicity, language and payer, and, where relevant, skin type and body habitus. Ask for results on your scanner or device vendors, EHR and protocols.
  • Local validation. Before go-live, run the tool on your own recent cases and compare it to ground truth. Agree in the contract on what counts as passing.

Workflow and people

  • Workflow cost. Count clicks, alerts per clinician per shift, minutes added per case, and who acts on the output. A strong tool that nobody acts on has no value.
  • Integration. Native EHR or PACS integration or a separate window? Single sign-on? Where do results land?
  • Human oversight. Who has the final say, how are disagreements with the tool handled, and how do you guard against over-reliance? Train users on known failure modes, not just features. Explanations matter most for risk scores and decision support.

After go-live

  • Monitoring. Who checks for drift, how often, and what triggers a pause? Re-check after vendor updates, scanner or EHR changes, and shifts in your patient population. Agree in advance on the results that would retire the tool.
  • Model updates. Version control, advance notice of changes, and the right to stay on a version you validated.

Data, liability and money

  • Data rights and security. BAA, where data is stored and processed, encryption, retention, SOC 2 or HITRUST, whether your data trains the vendor's models, the list of subcontractors that touch patient data (including any large-language-model provider), and whether data can be deleted.
  • Liability and patient notice. Who is liable when AI-assisted care goes wrong is still unsettled, and responsibility currently stays with the clinicians and organizations using the tool. Ask your malpractice carrier, and cover indemnification, audit rights and incident reporting in the contract. Joint Commission and CHAI guidance says patients should be told when AI directly affects their care, with consent where relevant.
  • Money and exit. Pricing model, any real payment pathway, the return you will measure, termination for performance failure, data return, and no auto-renewal without a performance review.

Privacy, liability, cost and monitoring

The questions committees most often leave until the contract stage. Settle them before you sign.

Privacy and patient consent

  • Recording. Ambient tools record conversations, and some states require every party to agree to a recording. Ask patients, document their answer, and give them an easy way to say no.
  • Patient notice. Tell patients in plain language when AI directly affects their care, as the Joint Commission and CHAI guidance recommends.
  • Data outside HIPAA. Patient-facing apps may fall under state consumer health-data laws or the FTC Health Breach Notification Rule instead of HIPAA. Ask which rules the vendor follows.
  • Minimum data. Confirm the tool receives only the data it needs, and how long audio, images and transcripts are kept.

Liability and contracts

  • Malpractice cover. Ask your carrier whether AI-assisted care is covered and what they expect from you, such as training records or documentation.
  • Indemnification. Look for vendor indemnity for software defects, data breaches and intellectual-property claims, and check the caps.
  • Audit and incidents. The right to audit, notice of security incidents and safety problems within set times, and cooperation with your investigations.
  • Documentation. Decide how clinicians record that they reviewed, edited or overrode the tool.

Return on investment and total cost

  • Total cost. License plus integration, IT staff time, training, hardware, support and renewal increases. Ask for a three-year total.
  • Measured return. Pick two or three measures before go-live, such as time saved, throughput, revenue captured or events avoided, and compare them with your own baseline, not vendor estimates.
  • Reference sites. Ask organizations of your size and EHR what they actually measured, not only whether they like the tool.
  • Hidden costs. Extra testing and visits caused by false positives, and the clinician time needed to review outputs.

Monitoring after go-live

  • Owner and schedule. Name one owner, review results at fixed intervals (for example monthly at first, then quarterly), and report to your AI or quality committee.
  • What to track. Accuracy against real outcomes on a sample, alert or usage volume, override rates, and results by patient group.
  • Triggers. Agree in advance on results that pause or retire the tool, and re-check after vendor updates and EHR or equipment changes.
  • Safety events. Report AI-related problems through your existing safety-event system, and tell the vendor.

What published studies show

In a 2026 analysis of 1,357 FDA-cleared AI devices, only 2.5% were linked to a registered prospective trial and only 0.2% had evaluated patient outcomes such as mortality or readmission.

PLOS Digital Health, 2026

The Joint Commission and the Coalition for Health AI (CHAI) published voluntary guidance in September 2025 built on seven elements: governance, privacy and transparency, data security, ongoing quality monitoring, safety-event reporting, risk and bias assessment, and education.

The Joint Commission, 2025

Further reading

Other evaluation guides

Last reviewed November 2026. Educational information, not medical, legal or regulatory advice.

No vendor reviewed, sponsored or edited this guide. Listing is free; no one can pay to change the order.