Auditing AI-generated notes

Ten signed notes per clinician, five scoring criteria, and a register the practice keeps.

Written by the aurii team

The audit reads a sample of signed notes, scores each one against five fixed criteria, and records the findings in a de-identified register.

A hand works through printed forms with a pen, coffee cup alongside

What the audit measures

The audit reads signed notes. A draft corrected before signing never entered the record.

Two steps produce a signed note: the draft the software wrote, and the review the clinician ran before signing. The register records which step a finding came from.

Record-keeping obligations attach to the clinician who signs, whether the entry was typed, dictated or drafted by software.

The sample

Ten signed notes per clinician per cycle.

  • Drawn per clinician. A practice-level average hides review discipline that varies by person.
  • Signed at least a week before the audit. The scorer reads the note cold.
  • Spread across consult types. Weak drafting clusters in fast multi-problem consults and where a carer is present.
  • Paired with the draft, where the scribe keeps both. The difference is what the review step caught.

What to read for

Score every note against the same five criteria.

  • Accuracy. Everything stated in the note happened in the consultation.
  • Completeness. Significant symptoms, examination findings, decisions, safety-netting advice and follow-up arrangements are present.
  • Attribution. The note separates what the patient reported from what the clinician observed, and identifies what a carer, interpreter or family member said.
  • Medication detail. Drug names, doses, frequencies, routes and changes to existing therapy match the chart.
  • Plan fidelity. Referrals, investigations, recall intervals and safety-netting advice match what was agreed with the patient.

Scoring and the register

Each criterion is scored met, minor issue, or significant issue. Significant is defined before the first cycle: an error that could affect care if it went unnoticed, such as a wrong dose, a wrong side, a missed red flag, or a plan that contradicts what was agreed.

A colleague scores the notes. Before the first cycle every scorer rates the same two or three notes, so the threshold for significant is shared.

Each register row carries the date, a de-identified reference to the note, the score for each criterion, and a one-line comment for anything scored other than met. Clinical content stays out, so a practice manager can read it without opening a patient record.

The register is the practice quality improvement evidence at accreditation.

Frequency

Run the audit monthly while the scribe is new, then quarterly. Practices with stable results across a year move to two cycles a year.

Repeat it after a new consult type, a new clinician, or a product update.

Acting on findings

The thresholds are set before the audit runs. Findings about draft quality go to the vendor with de-identified examples and the criterion they failed, and findings about scope go into the practice documentation policy.

  • Any significant medication finding. Acted on in the week it is found, and the record corrected.
  • A criterion carrying minor issues across several notes. A workflow change: a mandatory check before signing, or a change to which consult types use the scribe.
  • An isolated minor finding. Recorded and watched at the next cycle.

aurii and a note audit

aurii drafts the consultation note and the correspondence, and stops at the draft. Nothing leaves aurii until a clinician signs it. Clinical safety.

Edits made at review are recorded against the signature. Every action a practice user takes is written to a hash-chained audit log. How the chain works.

The method applies to any drafted note, whichever system produced it.

Correct the record with a dated addendum, and assess whether the error could have affected care. If it could have, follow the practice clinical incident process. Then read the rest of the sample for the same pattern.

Clinical audit is a recognised measuring outcomes activity for medical practitioners. Check the activity description with your CPD home before recording it.

This article is general information about auditing clinical documentation, not clinical or legal advice.

Start the free trial

Free 30-day trial. Create an account in the iPhone app or at app.aurii.com.au/signup. A card is added at signup on Stripe's checkout page; nothing is charged until the 30 days end.

hello@aurii.com.au