Buyer's guideHealthcare
Evaluating an ambient AI scribe: note quality, consent, EHR fit and a pilot plan
An ambient AI scribe records a consultation, transcribes it and drafts a clinical note for the clinician to edit and sign. Evaluating one well means sampling real notes against a structured error taxonomy, checking consent and audio retention, confirming how drafts reach the EHR, testing across accents and languages, and measuring documentation workload against your own baseline rather than vendor claims. This guide gives the questions, a review method and go or no-go criteria.
On this page
- What is actually being bought when you license a scribe
- An error taxonomy clinicians can apply to sampled notes
- Questions to put to every ambient scribe vendor, including us
- Consent, recording notices and where the audio goes
- An eight-week pilot that produces a defensible decision
- Criteria for scaling, extending or stopping the pilot
- Questions and answers
- Sources
What is actually being bought when you license a scribe
Behind the microphone are three components: speech recognition with speaker separation, a language model that turns the transcript into a structured note, and an integration that places the draft where the clinician signs. Some products add patient instructions, referral letters, order suggestions or billing codes. Each addition changes the risk profile, so evaluate the features you will switch on, not the full brochure.
The clinician who signs the note remains responsible for it. That makes the review burden the real product: a scribe that drafts quickly but needs careful line-by-line checking may save less time than one that drafts shorter notes clinicians trust. Your evaluation has to measure both draft quality and the editing effort it creates.
An error taxonomy clinicians can apply to sampled notes
Reviewers compare the draft with the audio or transcript and tag each error with one type and a severity.
| Error type | What it looks like | Why it matters | How to find it |
|---|---|---|---|
| Omission | A symptom, allergy, safety-net advice or plan item discussed but missing | Missing information is invisible on reading | Reviewer listens first, then checks the note against their own list |
| Addition | Findings, history or advice that were never said | Plausible fabrications can enter the record | Trace every examination finding to the audio |
| Misattribution | A relative's history recorded as the patient's, or a question recorded as a statement | Changes clinical meaning without looking wrong | Oversample visits with carers, interpreters or several speakers |
| Value error | Wrong dose, laterality, duration or number | Small text errors with large consequences | Check every number and side against the audio |
| Over-interpretation | The note states a diagnosis or severity the clinician did not reach | Shifts clinical and billing meaning | Compare assessment wording with what the clinician said |
Grade each error as no harm, potential minor harm or potential serious harm. A single potential serious harm error in the pilot sample should trigger a safety review, whatever the overall error rate.
Questions to put to every ambient scribe vendor, including us
Consent, recording notices and where the audio goes
Recording a consultation is a privacy event before it is a documentation one. In the US, some states require every party to consent to recording a confidential conversation; California's Penal Code section 632 is a commonly cited example2. Many organizations therefore ask for verbal consent at the start of every recorded visit, record that consent in the chart and give patients an easy way to decline without affecting their care.
Under HIPAA the vendor is normally a business associate, so the agreement should cover permitted uses, safeguards, breach reporting and return or destruction of data at the end of the contract3. Push for the shortest audio retention the product can work with, and be explicit about whether de-identified data may be used to improve the vendor's models.
In England, NHS England's guidance on AI-enabled ambient scribing products expects deploying organizations to complete clinical safety work under the DCB0160 standard and carry out a data protection impact assessment before use1. Whether a particular product, or a feature such as summarization or coding, is a medical device is a question to settle with the supplier and your regulatory advisers against current MHRA guidance.
An eight-week pilot that produces a defensible decision
Capture a baseline before anyone records
Pull documentation time, time spent in notes outside scheduled hours and note completion lag from EHR audit logs for the pilot clinicians and a comparison group.
Select a deliberately mixed cohort
Include enthusiasts and skeptics, several specialties, clinicians with different accents and clinics that use interpreters or see patients speaking other languages.
Agree the review sample and taxonomy
Fix how many notes each clinician will have reviewed weekly, who reviews them and the error types and severity grades in advance.
Run weekly reviews and a safety huddle
Reviewers score sampled notes against audio; the huddle looks at every potential harm error and any patient complaint or consent problem.
Remeasure workload and gather feedback
Repeat the audit-log measures, survey clinicians on editing burden and ask a sample of patients how the recording felt.
Take the go or no-go decision
Present results to the governance committee against criteria agreed before the pilot, including subgroup results.
Criteria for scaling, extending or stopping the pilot
- If
No potential serious harm errors, editing burden acceptable to most clinicians and workload down against baseline.
ThenScale in waves, keeping a smaller ongoing review sample.
Quality can drift with product updates, so review never drops to zero.
- If
Quality is good overall but poor for one language, accent or specialty.
ThenScale only where it works and restrict use elsewhere until the vendor fixes it.
Rolling out unevenly is better than shifting errors onto particular patients.
- If
Clinicians like it but audit logs show no change in workload.
ThenExtend the pilot with a sharper look at editing time and note length before buying at scale.
Satisfaction alone does not justify the license and review cost.
- If
Any unresolved consent, retention or contract gap.
ThenPause recording until it is closed.
Privacy failures are hard to undo once audio exists.
Questions and answers
Do patients have to agree to an ambient scribe recording their visit?
It depends on jurisdiction and your policies. Some US states require every party's consent to record a confidential conversation2, and organizations in England deploying these tools are expected to follow NHS England's guidance, including a data protection impact assessment1. Many organizations ask for verbal consent at each visit, document it and offer an easy opt-out. Take advice on the law where you practice.
Who is liable for errors in an AI-drafted note?
The clinician who reviews and signs the note is responsible for its content under the usual professional standards, and the organization remains accountable for the tools it deploys. Contracts can allocate some risk to the vendor, for example for security failures, but rarely shift clinical responsibility. This is not legal advice; involve your risk and legal teams before scaling.
Can an ambient scribe code encounters for billing?
Some products suggest diagnosis and procedure codes from the conversation. Treat these as coding support that a clinician or coder confirms, monitor for patterns that change coding levels, and include them in your billing compliance reviews. Code suggestions also extend what the product does beyond documentation, so confirm the vendor's regulatory position for that feature.
Sources
- Guidance on the use of AI-enabled ambient scribing products in health and care settings — NHS England · checked 10 October 2026
- California Penal Code section 632 — California Legislative Information · checked 10 October 2026
- 45 CFR § 164.504 Uses and disclosures: organizational requirements (business associate contracts) — eCFR · checked 10 October 2026