CIFoundry

How to evaluate radiology AI

Radiology has more FDA-cleared AI than any other field, and most of it reaches market through 510(k). That makes the details of each clearance, and your own testing, especially important.

A triage tool that reorders the worklist, a detection aid that marks findings, software that reconstructs images and a tool that drafts report text each fail in different ways. Use the section that matches the tool.

Last reviewed November 2026 · Independent and vendor-neutral

What does the tool you’re evaluating do?

Jump to the matching checklist. Some answers lead to a different guide.

Radiologists usually want

  • Accuracy on their own scanners and protocols
  • Few false positives in the worklist
  • Results inside PACS with no added clicks
  • Measurements and text they would sign

Imaging leaders usually want

  • Faster turnaround for critical findings
  • Reads per radiologist
  • One platform contract rather than many point solutions
  • Governance aligned with ACR ARCH-AI and monitoring such as ACR Assess-AI

Urgent-finding triage and notification

Applies if the tool screens studies for urgent findings, such as stroke, brain bleed or pulmonary embolism, and moves them up the worklist or alerts the care team.

Questions to ask

  • What are sensitivity and specificity on our scanner models and protocols?
  • How many false alerts per day should we expect at our volume?
  • Does it actually change read order and time to treatment, and can you show data from a site like ours?
  • Who receives mobile alerts (stroke team, ED), and is that part of the cleared use?

Test it locally

  • Measure time from scan to read and to treatment before and after go-live, and audit every missed case in the first months.

Red flags

  • No scanner or protocol breakdown
  • Turnaround claims without before-and-after data

Detection and characterization aids

Applies if the tool marks possible findings or rates how suspicious they are, for example lung nodules, fractures or lesions on screening mammograms.

Questions to ask

  • What did the FDA test set include: number of cases, sites, scanner vendors? Which earlier device did the clearance rely on?
  • How does it perform by age, sex, breast density or body habitus, as relevant?
  • What failure modes are known? An algorithm with a blind spot misses the same kind of case on every study, not just once.
  • How do you guard against radiologists over-trusting the marks, or under-reading studies with no marks?

Test it locally

  • Run it silently on several hundred recent studies with known outcomes, per scanner model, before switching it on.

Red flags

  • No subgroup data
  • Results only in a separate viewer
  • No monitoring dashboard

Measurement and quantification

Applies if the tool measures structures or lesions, for example organ volumes, brain atrophy, lesion size or change over time.

Questions to ask

  • How do measurements compare with your radiologists, and how repeatable are they on the same patient?
  • Are results consistent across scanner models and software upgrades, so follow-up comparisons are valid?
  • Can radiologists correct a measurement, and is the correction kept?

Test it locally

  • Re-check measurements after any scanner or protocol change.

Image reconstruction and acquisition

Applies if the tool changes how images are made, for example deep-learning reconstruction, noise reduction, lower-dose CT or faster MRI.

Questions to ask

  • Could it remove or create detail, such as small lesions, and how was that tested?
  • Which scanners and sequences is it cleared for?
  • What dose or scan-time reduction have similar sites achieved while keeping diagnostic quality?

Test it locally

  • Have radiologists compare standard and AI-processed images side by side on cases with known subtle findings.

Report drafting

Applies if the tool drafts findings or impressions for the radiologist to edit and sign.

Questions to ask

  • Is it FDA-cleared for this use, or positioned as a non-device writing aid? What does its intended use say?
  • How often does a draft contain a finding not in the images, or miss one that is?
  • Does it keep your follow-up recommendation standards and structured templates?
  • Which large-language-model provider is used, and where are images and reports processed?

Red flags

  • No measured error rate
  • Drafts that can be signed without review

What published studies show

In the MASAI randomized trial in Sweden, AI-supported mammography screening detected more cancers than standard double reading (6.4 vs 5.0 per 1,000) and cut screen-reading workload by 44%, without a significant rise in false positives. The interval-cancer result was still being assessed at publication.

Lancet Digital Health, 2025
Whatever the tool does, the 12 core questions (evidence, data rights, liability, monitoring, contracts) also apply.

Further reading

Browse radiology tools

Every listing shows regulatory status, published evidence and named health-system users where public.

Other evaluation guides

Last reviewed November 2026. Educational information, not medical, legal or regulatory advice.

No vendor reviewed, sponsored or edited this guide. Listing is free; no one can pay to change the order.