Letter to the Editor - Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study (Preprint)

UNSTRUCTURED We have several comments on the recent publication of [1], in which repeated testing of four symptom assessment applications with clinical vignettes was carried out to look for “hints of ‘non-locked learning algorithms’”. As the developer of one of the symptom assessment applications studied by [1], we are supportive of studies evaluating app performance, however there are important limitations in the methodology of the study. Most importantly, the methodology used in this study is not capable of addressing its main objective. The approach used to look for evidence of non-locked algorithms was the quantification of differences in performance using three ophthalmology vignettes, first in 2018 then in 2020. This methodology, although highly limited due to the use of only three vignettes in one medical specialism, could be used to detect changes in app performance over time. It however cannot be used to distinguish between non-locked algorithms and the manual updating of the apps’ medical intelligence, through the normal process of manual release of updated app versions.