Accuracy of an Artificial Intelligence System for Cancer Clinical Trial Eligibility Screening (Preprint)
BACKGROUND Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process. OBJECTIVE This study aims to evaluate the ability of an AI clinical decision-support system (CDSS) to identify eligible patients for a set of clinical trials. METHODS This study included the de-identified data from a cohort of breast cancer patients seen at the medical oncology clinic of academic medical center between May and July 2017 and assessed patient eligibility for four breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Fleiss’ kappa, and significance of differences was determined by Wilcoxon signed-rank test. RESULTS Three hundred eighteen breast cancer patients were included. Interrater reliability for manual screening was 0.64, indicating substantial agreement. The overall accuracy of breast cancer trial-eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%. CONCLUSIONS The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offers the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.