Accuracy moves with the population: what an AI-ECG study teaches human readers

An algorithm called EchoNext was trained on hospital ECGs to detect structural heart disease — valve lesions, ventricular dysfunction, the things you normally need an echocardiogram to see. In hospital cohorts it discriminated well, with an area under the curve of 83%.

Researchers then took it into people's homes. PREVUE-VALVE recruited community-dwelling adults aged 65 to 85 and gave each of them an in-home ECG and an echocardiogram. Among the 2,402 participants analysed, the same model scored an AUC of 71%.

Nothing about the model changed. No code was edited, no weights retrained. What changed was who was being tested — and that turns out to be enough.

First: why is structure visible on an ECG at all?

It is worth pausing on what the model was even doing. An ECG records electrical activity, not anatomy. But electrical activity is shaped by the muscle it passes through, so structure leaves fingerprints on the trace: increased voltage when the ventricular wall thickens, altered P wave morphology when an atrium enlarges, low voltage when fluid or fat sits between heart and electrode, conduction delay when the chamber geometry changes.

Human readers have used these signals for a century, and the standard criteria for chamber enlargement are well described. The catch is that they are specific but insensitive: a positive voltage criterion means something, but a normal one rules out very little. An AI model is doing a more sophisticated version of the same trick — finding subtler combinations of the same physics.

The spectrum effect

In the hospital cohorts, 43% of patients had structural heart disease. In the community cohort, 8% did. And the disease that was present was different in kind, not just in quantity: more moderate tricuspid regurgitation, less systolic heart failure, milder phenotypes throughout.

That is what is meant by disease spectrum. A hospital population is enriched — people are there because something is already wrong, and by the time they arrive the abnormality is usually well developed. The distinction between health and disease is stark, and any test looks good at making it. In a community population the same distinction becomes blurred, and every test gets harder.

The study's authors did something worth noting: they propensity-matched to account for the differences, and the performance gap narrowed but did not disappear. Meanwhile the model performed consistently across several different hospital cohorts. Both findings point the same way — this was not a broken model or a bad dataset, it was the population itself.

Prevalence, and the number nobody teaches you

Sensitivity and specificity are properties you can quote from a paper. Neither answers the question a clinician actually has, which is: this test is positive, what is the chance the disease is really there?

That is positive predictive value, and it depends on how common the disease is. Take an illustrative test with 80% sensitivity and 80% specificity, applied to 1,000 people:

Identical test. Identical sensitivity and specificity. A positive result means something quite different depending on who walked through the door. In the low-prevalence setting most positives are false, and that is arithmetic rather than a flaw in the test.

The PREVUE-VALVE data show this directly. Performance improved in subgroups where disease was more likely — an AUC of 79% in participants whose ECG was already abnormal, and 76% in those with impaired health status.

What this changes about how you read

The lesson is not about algorithms. Your own pattern recognition is a diagnostic test, and it obeys the same rules.

• Form a pre-test probability before you look. Who is this, how old, what are the symptoms, what is the story. The trace is evidence to be weighed against that, not a verdict that replaces it.
• The same finding carries different weight in different people. Borderline voltage criteria in a 25-year-old athlete and in an 80-year-old with breathlessness are the same millimetres and completely different information.
• Beware the screening ECG. Applying pattern recognition to people with no symptoms is the low-prevalence column of that table. Most of what you flag will be false, and each false positive buys someone an investigation they did not need.
• A normal ECG rarely excludes structural disease. That was true before AI and remains true after. The trace is insensitive for structure; echocardiography exists for a reason.

Reading the paper honestly

Two things deserve stating plainly. The inventors of the algorithm are among the authors and may receive royalties or licensing fees related to it — which is disclosed in the paper, and is the kind of detail worth checking in any study of a commercial diagnostic. And the finding is not that AI-ECG failed. An AUC of 71% for detecting structural heart disease from twelve leads and no imaging is still a real signal. The finding is narrower and more useful: performance measured in one population does not transfer unexamined to another.

Which is a good discipline for a human reader too. The accuracy you have on exam traces, hand-picked to show a finding, is not the accuracy you will have at three in the morning on an unselected patient. Same reader, different spectrum.

Educational use only. This article is a teaching resource. Nothing here is a diagnostic device, and nothing here replaces certified clinical training, local protocols or senior clinical advice. Never use it to make a decision about a real patient.