All studies

Medical AI / Reproducibility · APTOS 2019 · IEEE SoutheastCon 2027 · Oral presentation

When 98% accuracy is really 83%: leakage in eye-screening AI

Deep learning models for diabetic retinopathy screening often report near-perfect accuracy on the public APTOS 2019 dataset. We audited the dataset for duplicate photographs, read the released code of 12 studies line by line, and reran three of them. Done the way a screening device would actually see photos, one at a time, a published 98.2% model fell to 83.2% and missed about eight times as many referable cases.

98.2% → 83.2%five-grade accuracy, published protocol vs. one photo at a time
~1 → ~8referable eyes missed per 100
270images that are repeated photographs (7.4%)
12studies’ code read line by line

Unwind the leakage, one fix at a time

Pick how the data were split and how test photos were fed to the model. The published setup is in the top left. A real screening device sees one photo at a time.

Split
Test order

Out of 100 eyes that need referral, how many are missed?

Referable diabetic retinopathy is grade 2 or higher. Each circle is one eye that should be sent to a specialist.

7.4% of the dataset is repeated photographs

Each dot is one of the 3,662 APTOS 2019 images. Shuffle a random split and count how many test photos already have a copy in the training set.

The simulation uses the audit’s duplicate group sizes and per-grade concentration; dot positions are illustrative. Across 500 random 80/20 splits the audit measured 6.0% (range 4.6–7.4%).

Picking the best checkpoint on the test set

Accuracy gained by choosing the saved checkpoint on test data instead of a held-out validation set (mean over three seeds, 95% CI).

What the released code actually does

We coded five leakage checks against each study’s code at a fixed commit. Pick a check to highlight the studies that fail it.

Inside the study

Question · Data · Methods · Findings · Checklist · Limits
Question

How much of the accuracy reported for diabetic retinopathy classifiers on APTOS 2019 comes from data leakage, and what does a screening model look like when it is evaluated the way a device would use it?

Why it matters

APTOS 2019 was released to help screen for diabetic retinopathy in rural areas where specialists are scarce. Because its official test labels were never released, every study builds its own split, and reported five-grade accuracies range from the low 80s to above 98%. If a benchmark number is inflated, a model chosen for a screening device on that basis may miss more disease in practice than its paper suggests.

Data

The 3,662 labeled fundus photographs of the APTOS 2019 Blindness Detection set, graded 0 (no retinopathy) to 4 (proliferative). For the literature, a PRISMA 2020 search of GitHub, Papers with Code, arXiv, Google Scholar, publisher sites and Europe PMC returned 372 records; 12 studies published 2020–2025 developed and tested on APTOS 2019 and released runnable code.

Methods

An audit tool compared all 6,703,291 image pairs using byte and pixel hashes, 256-bit perceptual hashes and thumbnail cross-correlation, with manual review of sampled pairs. Each included repository was coded on five leakage items at a fixed commit, with an independent second rater (κ = 0.97). Three studies (DiffMIC, nnMobileNet and a MobileViT + graph network, GCN) were rerun with three seeds under their released protocol and under corrected protocols, with instrumentation only.

Findings

The audit found 131 groups of repeated photographs covering 270 images (7.4%), 37 of them with conflicting grades; random splits leave a training copy for about 5–6% of test images. Choosing checkpoints on the test set added 0.8–2.2 accuracy points. For GCN, oversampling before the split plus grade-sorted test batches accounted for 16.6 points (98.2% under the published protocol versus 81.6% with a duplicate-aware split in random order). One image at a time, the corrected model reached 83.2%, and referable-DR sensitivity fell from 99.0% to 92.2%.

Checklist for evaluating screening models

1) Split by duplicate group and say how conflicting grades were handled. 2) Use a held-out validation set for every choice made after looking at data. 3) Create augmented or oversampled copies only after splitting, inside training. 4) Report results one image at a time when predictions depend on the batch. 5) Release split IDs, the selection rule and the evaluation script. 6) Report the mean and spread over several seeds. 7) Report referable-DR sensitivity and specificity under the inference mode the device will use.

Limitations

Three seeds per configuration give wide confidence intervals. GCN’s oversampling parameters and nnMobileNet’s split were not released and were reconstructed. All experiments used one dataset, and these figures are not estimates of performance on a deployed device. The duplicate detector does not reliably find crops, so duplicate counts are lower bounds.

Recognition

Accepted for oral presentation at IEEE SoutheastCon 2027 (April 2027). The audit tool, duplicate groups and corrected split files are being released with the paper.

Sources

Images are available from Kaggle under the competition rules and are not redistributed here.