A personal idea, not affiliated with or endorsed by any organization. Synthetic data only: results are scored against the generator's truth, not clinical performance, and costs use illustrative prices.

PerioBench · executive pitch · the dental record (EDR) · October 2026

The diagnosis is in the dental record.

Nobody can count it.

PerioBench: can more accurate diagnosis from the dental record improve care for gum disease, for millions of patients? Checked against gum specialists, built and tested end to end on synthetic records.

Health Transformation Lab

Working name · synthetic data only

John Bowyer · personal project, not affiliated with any organization

Gum disease is one of the most common chronic diseases in adults, and it is written down at almost every dental visit. But it is written down in pieces: a checkbox here, a billing code there, measurements, and a free-text note. Nobody can count it across clinics, so nobody can study it at scale, link it to diabetes, or see whether care is working. PerioBench asks one question: can a more accurate diagnosis from the dental record improve care for people with gum disease, so it is found earlier, treated, followed over time and managed alongside diabetes? Agreement with gum specialists is how we check the diagnosis is accurate; better patient care is the goal. Everything shown is on synthetic records, built so the right answer is known; it is a rehearsal of the real study, not a result on real patients. This is a personal project, not any organization's plan.

The problem

Common, linked to diabetes, and invisible in the data

US adults 30+

42%

have gum disease (periodontitis); 7.8% have the severe form.

Diabetes

Both ways

Gum disease and diabetes are linked: each is associated with worse control of the other.

In the record

7 places

where a dental chart holds evidence of gum disease: notes, forms, codes, measurements. They often disagree.

Dental records rarely hold one reliable diagnosis. The status has to be worked out from the evidence, the way a specialist would, before anyone can count it.

NHANES 2009–2014 (Eke et al., J Am Dent Assoc 2018) · EFP and IDF consensus report (Sanz et al., 2018) · seven evidence types: this repo · 2

Three facts. One: about 42 percent of US adults aged 30 and over have periodontitis, gum disease that destroys the attachment and bone around teeth, and 7.8 percent have the severe form; that is the national survey, NHANES 2009 to 2014, published by Eke and colleagues in 2018. Two: gum disease and diabetes are linked in both directions; the European Federation of Periodontology and the International Diabetes Federation ask doctors and dentists to manage the two together. Three, and this is the one we can fix: in the dental record the diagnosis is rarely written down once, reliably. It is spread over seven kinds of evidence: the clinician's note, an exam form with a score and a checkbox, the procedure codes done and planned, the diagnosis codes on those procedures, and the gum measurements. They often disagree. So before anyone can count gum disease across clinics, or link it to diabetes in the medical record, someone, or something, has to read the chart and work out the status. If asked for sources: Eke PI et al., Periodontitis in US adults: NHANES 2009-2014, JADA 2018, https://doi.org/10.1016/j.adaj.2018.04.023; Sanz et al., J Clin Periodontol 2018, https://doi.org/10.1111/jcpe.12808. The seven evidence types are this study's design; see the evidence rules page.

What we built

Read the whole record, then prove it against specialists

Synthetic patients

8 clinics, visits over time: notes, forms, codes, gum measurements, with fake names in the notes

Privacy first

Names, dates and IDs removed; a final check holds back any note with one left

A common format

The record in OMOP, the format medicine already uses, so dental links to medical

Rules and AI read it

Rules combine the whole record; AI models read the note alone

Specialists check it

Periodontists label the same 240 charts, each on their own (simulated reviewers so far)

The acceptance test

AI passes only if it agrees with them as often as they agree with each other

Built to run on a real extract unchanged: the same code, tests and dashboards, once a de-identified dataset and its approvals exist.

Synthetic data only · Health Transformation Lab · the dental record (EDR) · 3

This is the whole pipeline, and it runs end to end today on synthetic data. One: we generate synthetic patients across 8 clinics, visit after visit, with every kind of evidence and fake names in the notes, so the privacy step is tested too. Two: names, dates and IDs are removed, and a final check holds back any note that still has one. Three: the record goes into OMOP, the common data format medicine already uses, which is what lets a dental finding sit next to a diabetes diagnosis. Four: rules combine the whole record, and AI models read only the free-text note, for the many charts where the structured fields are missing. Five: gum specialists label the same 240 charts, independently; until real specialists do, simulated reviewers stand in so the reliability code can be tested, and their numbers are never shown as results. Six: the acceptance test. An AI method passes only if it agrees with the specialists at least as often as they agree with each other; that is the standard used by the OHDSI community for validating phenotypes. Because it is synthetic, we know the right answer for every chart, so we can test the test. If asked about the platform: Databricks and Unity Catalog, MLflow for models, Postgres and Lakebase for OMOP; Python and R give the same numbers.

What it shows so far (synthetic)

The whole record answers it. The note alone nearly does.

Whole record, rules

234 of 240

charts given the right status, combining the evidence, gum measurements first.

Note only, trained AI

219 of 240

right from the free-text note alone (ModernBERT, with a rule for the most severe stage).

Note only, medical AI as is

185 of 240

for MedGemma 1.5 4B untrained, with the same stage 4 rule: about as good as keyword rules (189).

Gum measurements alone

193 of 240

right, but 45 charts had no measurements to read: one piece of evidence is not enough.

Training beats size: a small model trained on dental notes beat a bigger medical AI used as is. Synthetic notes are easier than real ones, so expect the gaps to widen.

Synthetic charts, scored against the generator's truth: not clinical performance · real model runs, CPU, 2026-10-26 · 4

These are real model runs, on synthetic charts, scored against the generator's truth; they are not clinical performance, and agreement with simulated reviewers is a dry run we do not show. Four numbers out of 240 review charts. Rules that combine the whole record, reading the gum measurements first, got 234 right. Reading only the free-text note, a small model trained on dental notes, ModernBERT with a keyword rule for the most severe stage, got 219. A bigger medical AI, MedGemma 1.5 4B, used as is without training and with the same stage 4 rule, got 185, about the same as simple keyword rules at 189; alone it gets about 182. And any single piece of evidence falls short: the gum measurements alone got 193 right, because 45 of the 240 charts had no measurements at all. Two lessons: combine the evidence, and train on the domain. If asked about uncertainty: every number has a 95 percent bootstrap interval on the full results page; on 240 charts, many methods cannot be told apart. If asked whether MedGemma can improve: few-shot and fine-tuned runs are next, on a GPU. If asked why synthetic: no real patient data may be used here; synthetic data lets us build and test everything, including the privacy checks, before any approvals.

Why it pays

Computers are cheap. Specialist hours are not.

By hand

33,333 h

to review a million visits, one person at 2 minutes a chart.

By software

Hundreds of $

of computing for the same million notes, even for the largest models priced (simulated).

To prove it

~31 h

of specialist time: 233 charts, 4 reviewers, to run the acceptance test.

The case is not the computing price: it is replacing review hours, once the software passes the acceptance test with a margin people accept.

Illustrative prices and assumed throughput (a dry run) · review pace from the labelling protocol, 1–3 minutes a chart · 5

The economics. Reading a million visits by hand, at the protocol's pace of about 2 minutes a chart, is 33,333 hours of a specialist's time. The software does the same million notes for tens to hundreds of dollars of computing, even for the largest models we priced; those are illustrative prices and assumed speeds, a dry run, not quotes. Proving the software is good enough is small: about 31 specialist hours, 233 charts each read by 4 reviewers. So the argument is not that computers are cheap. It is that specialist hours are the bottleneck, and the software can replace most of them, but only after it passes the acceptance test. If asked for a dollar figure: at an illustrative 100 dollars an hour, validation is about 3,100 dollars and reviewing everything by hand about 3.3 million. If asked about errors: the study also counts unsafe errors, such as missing severe disease, and reports them separately.

The ask

From synthetic to real

What we do

Run the same pipeline on a de-identified extract, inside its owner's environment and approvals
Four gum specialists label the validation charts, each on their own (~8 hours each)
Publish the acceptance-test result, good or bad, with the code

If it works

Gum disease countable across clinics, in the format medicine uses
The medical-dental link, such as diabetes, studied at scale
A tested reader that checks AI answers against the record

The ask: a data partner willing to run it on a de-identified extract in its own environment, under its own approvals; four periodontists for about a day each; and a few days of GPU time. No partner has been contacted.

Health Transformation Lab · the dental record (EDR) · 6

The ask is small, because the build is done. We need three things. One: a data partner willing to run the pipeline on a de-identified extract, under its own approvals; the code is built to run unchanged on a real extract, and nothing leaves their environment. Two: four gum specialists, about a day each, to label the validation charts independently; that is the 31 hours on the previous slide. Three: a few days of GPU time, for the larger models. In return: we publish the acceptance-test result, good or bad, with the code. If it works, gum disease becomes something you can count across clinics, in the same format as the medical record, so the link with diabetes can be studied at scale; and the same reader becomes a way to check AI answers against the record, which is the other path on the landing page. No partner has been contacted, and nothing here is any organization's plan. If asked what could go wrong: real notes are harder than synthetic ones, and the AI may not pass; that is still a publishable answer.

Health Transformation Lab · the dental record

Drill-down

Backup slides for questions. The full science is on the study page.

The order the evidence is read in, and why it changes the answer
How we know it is right: the acceptance test, and an AI that checks AI
Privacy first: removing names before anything reads a note
How this path feeds the other: test questions with known right answers

The order matters

Same evidence, read in a different order: a different answer

Gum measurements first

234 of 240

right: then the note, exam form and codes fill the gaps (the first rule set, v0).

Exam form first

173 of 240

right (72%): the study design's order, exam form, diagnosis codes, note, then measurements.

Orders to test

24

four sources can be ranked 4×3×2×1 ways; each ranking is scored against the specialists.

So the order is a finding, not a setting: the ranking that agrees best with the specialists is the one to keep. Procedure codes are a rough guess from the treatment, so they are compared, not used.

Synthetic charts, scored against the generator's truth · 2017 AAP/EFP classification · rules and worked examples: docs/evidence_rules.md · 8

A dental chart holds seven kinds of evidence: the clinician's note; the exam form, with a gum score and a status checkbox; the procedure codes done and planned; the diagnosis codes on those procedures, done and planned; and the probing chart of gum measurements, turned into a stage with the 2017 American Academy of Periodontology and European Federation of Periodontology worksheet. Procedure codes are only a rough guess from the treatment, so they are compared, not used. When the other sources disagree, the phenotype takes the first one, in a fixed order, that gives a status. The order matters a lot. Reading the gum measurements first, the first rule set, gets 234 of 240 synthetic charts right; it is the 234 on the results slide. The study design's order, exam form first, then diagnosis codes, the note, and the measurements last, gets about 173, 72 percent. On synthetic charts the measurements are the generator's own evidence, which flatters reading them first; on real charts the answer may differ. Four sources can be ranked 24 ways; each ranking is scored against the specialists, and the one that agrees best is kept. A radiograph reading can only raise the stage.

How we know it is right

The acceptance test, and an AI that checks AI

The bar

As good as us

AI passes only if it agrees with specialists as often as they agree with each other.

AI as checker

83%

of charts where a general-purpose AI's own reading matched the answer key; 53% on hard notes.

Catching mistakes

26%

of the best note model's mistakes caught by that AI checker: mostly near-misses, one stage off.

So on real notes, people still set the answer key: the AI checker spots weak methods, but cannot yet stand in for specialists on good ones.

Synthetic charts; LLM judgments checked against the generator's truth · the bar: OHDSI 2023 plenary on case validation; npj Digit Med 2025 · 9

How do we know an answer is right? On real records there is no answer key, so the acceptance test is the bar: an AI method passes only if it agrees with the specialists at least as often as they agree with each other, the standard from the OHDSI 2023 plenary on case validation, with the LLM review work published in npj Digital Medicine in 2025. We also tested whether a general-purpose AI, a large language model, could stand in for the answer key. Reading each note itself, it matched the answer key on 83 percent of charts but only 53 percent of hard notes, the ones that leave out what the answer key is built from. Asked whether a method's answer was right, it caught most mistakes of weak methods, but only 26 percent of the best note model's, which are near-misses such as one stage off. So it cannot replace specialists yet. These are AI judgments on synthetic notes, checked against the generator's truth. If asked about statistics: kappa for agreement beyond chance, 95 percent bootstrap intervals over charts, and unsafe errors counted separately.

Privacy first

No note reaches a model with a name left in it

Identifiers missed

115 of 2,714

by the best combination of tools on 400 synthetic notes; the final check holds those notes back.

Dental words damaged

41 of 2,462

terms that look like names or codes (Glickman, Gracey 11/12) removed by mistake.

Methods compared

15

pattern matching, single AI tools, and combinations, with a dental-word guard.

Two numbers that pull against each other: remove too little and patients are at risk; remove too much and the note the AI reads is damaged.

Synthetic notes with fake names and dates; also tested on public benchmark sets · Health Transformation Lab · the dental record (EDR) · 10

How the two paths connect

The record gives the test its right answers

This path

The record

Synthetic patients whose right answer is known, in the format medicine uses.

The bridge

Test questions

Case questions written from those patients, with the right answer already scored.

The other path

The lab

AI tested where medicine meets dentistry and access to care, in public.

One harness, narrow to broad: the same models, runners, harm grades and leaderboards score AI first on gum charts, then on case questions, then on broader medical-dental ones.

The lab's pitch: docs/pitch/ · Health Transformation Lab · the dental record (EDR) · 11