Evaluation packet
GatorTron
University of Florida & NVIDIA · NLP & EHR · Other NLP
Prepared November 22, 2026 from the public listing on CIFoundry. It shows what the public record says and what to ask the vendor. It is not a recommendation or an endorsement.
1. What the public record shows
- Regulatory status
- Not on record
- Published evidence
- Not on record
- Named health-system users
- Not on record
- Integrations
- Not on record
- Data inputs
- Not on record
- HIPAA BAA
- Not on record
- SOC 2 / HITRUST
- Not on record
- Deployment model
- Not on record
Integrations, data inputs, BAA, SOC 2, HITRUST and deployment model are reported by the vendor and not checked by us. Confirm them in writing.
2. Evaluation checklist for your role
Each question shows what the public record says: on record (with its source), reported by the maker, or not on record. It is a list of things to check, not a score.
For every reviewer
Administrator & finance
Physician
IT & security
Compliance & risk
Your ticks and notes stay in this browser. Nothing is sent to us.
3. Gaps to raise with the vendor
Based on what is missing or worth confirming in this tool’s public record.
- No regulatory status is listed. Ask whether the tool is regulated by the FDA as a medical device and, if so, for the clearance number and the exact cleared intended use.
- No published study is linked. Ask for peer-reviewed validation done outside the developer’s own sites, and for results on patients and equipment like yours.
- We found no public reports naming health systems that use it. Ask for reference sites similar to yours and contact them yourself.
- Will the vendor sign a HIPAA Business Associate Agreement?
- Does the vendor have a current SOC 2 Type II report or HITRUST certification you can review?
- Where is data stored and processed (cloud region, on-premises), and which subcontractors, including any large-language-model provider, touch patient data?
- How does it integrate with your EHR or imaging system, and where do results appear for the clinician?
- Is your data used to train the vendor’s models, and can it be deleted when the contract ends?
4. Questions for this kind of tool
From our Clinical documentation AI guide.
Change tool type:
Ambient scribes
Use this section if the tool records the patient visit and drafts the clinical note.
Questions to ask
- What share of notes are signed without edits, and how is that measured?
- What is the rate of critical errors (invented findings, wrong medication, wrong side) in the latest independent evaluation?
- How does it handle accents, non-English visits, interpreters, family members and noisy rooms?
- Does it support your specialty's visit types and templates (procedures, behavioral health, pediatrics)?
- Where is audio stored, for how long, and is it used to train models?
- What happens when a patient declines recording?
Test it locally
- Pilot for 30 to 60 days with a mix of specialties and clinician types, including heavy and light users.
- Measure documentation time, total EHR time and after-hours time from EHR audit logs before and during the pilot, not only surveys.
- Score a blinded sample of notes for quality (PDQI-9 is a common instrument) and count critical errors.
- Track E/M level distribution before and after, to catch both lost revenue and unintended upcoding.
Red flags
- Only self-reported time savings
- Audio kept indefinitely or used for training by default
- No way to measure how much clinicians edit
- No consent workflow that respects state recording laws
Chart summarization and search
Use this section if the tool reads the existing record and produces a summary, a pre-visit brief or answers to questions about the chart.
Questions to ask
- Does every statement in the summary link back to the source document so it can be checked?
- How does it handle conflicting or outdated information (old medication lists, resolved problems)?
- What does it do when information is missing: say so, or fill the gap?
- Which parts of the record can it see (outside records, scanned documents, notes from other specialties)?
Test it locally
- Have clinicians compare summaries against full chart review on a sample of complex patients and count omissions and errors separately. Omissions are harder to notice than errors.
Red flags
- No source citations
- Accuracy reported only on simple patients
Patient message replies
Use this section if the tool drafts replies to patient portal messages for clinicians or staff to send.
Questions to ask
- Does a human always review before sending, and does the system prevent unreviewed sending?
- How does it recognize urgent symptoms in a message and route them?
- Is the patient told that AI helped draft the reply?
- Can drafts be set to your reading level and languages?
Test it locally
- Review a sample of drafts for clinical accuracy, tone and missed red-flag symptoms, and measure whether reply time actually falls.
Red flags
- Auto-send options for clinical content
- No urgent-symptom escalation
EHR assistants and clinical question answering
Use this section if the tool answers clinical questions, suggests orders or acts as a general assistant inside the EHR.
Questions to ask
- Does it cite its sources, and are they current guidelines and literature?
- If it recommends diagnoses or treatments, is it regulated as decision support, and what is its stated intended use?
- Which large-language-model provider does it use, and does patient data leave your environment?
- How is it updated, and will you be told when answers may change?
Red flags
- Unsourced clinical answers
- No disclosure of the underlying model provider
5. Questions for every tool
The 12 questions that apply to every tool
Fit and evidence
- Intended use. Does the stated use match the patients, setting, user and decision you plan to use it for?
- Evidence level. From weakest to strongest: vendor-internal test, retrospective external validation, prospective "silent" run, prospective live study, randomized trial, proven change in outcomes. Ask which level the claims rest on and look for peer-reviewed external validation.
- Population and equipment. Ask for results by age, sex, race and ethnicity, language and payer, and, where relevant, skin type and body habitus. Ask for results on your scanner or device vendors, EHR and protocols.
- Local validation. Before go-live, run the tool on your own recent cases and compare it to ground truth. Agree in the contract on what counts as passing.
Workflow and people
- Workflow cost. Count clicks, alerts per clinician per shift, minutes added per case, and who acts on the output. A strong tool that nobody acts on has no value.
- Integration. Native EHR or PACS integration or a separate window? Single sign-on? Where do results land?
- Human oversight. Who has the final say, how are disagreements with the tool handled, and how do you guard against over-reliance? Train users on known failure modes, not just features. Explanations matter most for risk scores and decision support.
After go-live
- Monitoring. Who checks for drift, how often, and what triggers a pause? Re-check after vendor updates, scanner or EHR changes, and shifts in your patient population. Agree in advance on the results that would retire the tool.
- Model updates. Version control, advance notice of changes, and the right to stay on a version you validated.
Data, liability and money
- Data rights and security. BAA, where data is stored and processed, encryption, retention, SOC 2 or HITRUST, whether your data trains the vendor's models, the list of subcontractors that touch patient data (including any large-language-model provider), and whether data can be deleted.
- Liability and patient notice. Who is liable when AI-assisted care goes wrong is still unsettled, and responsibility currently stays with the clinicians and organizations using the tool. Ask your malpractice carrier, and cover indemnification, audit rights and incident reporting in the contract. Joint Commission and CHAI guidance says patients should be told when AI directly affects their care, with consent where relevant.
- Money and exit. Pricing model, any real payment pathway, the return you will measure, termination for performance failure, data return, and no auto-renewal without a performance review.
6. Notes and decision
Problem we want to solve and how we will measure it
Vendor answers and follow-ups
Local test plan and what counts as passing
Decision, owner and next step