Design and user experience testing of a polygenic score report: a qualitative study of prospective users

Background Polygenic scores—which quantify inherited risk by integrating information from many common sites of DNA variation—may enable a tailored approach to clinical medicine. However, alongside considerable enthusiasm, we and others have highlighted a lack of standardized approaches for score disclosure. Here, we review the landscape of polygenic score reporting and describe a generalizable approach for development of a polygenic score disclosure tool for coronary artery disease. Methods We assembled a working group of clinicians, geneticists, data visualization specialists, and software developers. The group reviewed existing polygenic score reports and then designed a two-page mock report for coronary artery disease. We then conducted a qualitative user-experience study with this report using an interview guide focused on comprehension, experience, and attitudes. Interviews were transcribed and analyzed for themes identification to inform report revision. Results Review of nine existing polygenic score reports from commercial and academic groups demonstrated significant heterogeneity, reinforcing the need for additional efforts to study and standardize score disclosure. Using a newly developed mock score report, we conducted interviews with ten adult individuals (50% females, 70% without prior genetic testing experience, age range 20–70 years) recruited via an online platform. We identified three themes from interviews: (1) visual elements, such as color and simple graphics, enable participants to interpret, relate to, and contextualize their polygenic score, (2) word-based descriptions of risk and polygenic scores presented as percentiles were the best recognized and understood, (3) participants had varying levels of interest in understanding complex genomic information and therefore would benefit from additional resources that can adapt to their individual needs in real time. In response to user feedback, colors used for communicating risk were modified to minimize unintended color associations and odds ratios were removed. All 10 participants expressed interest in receiving a polygenic score report based on their personal genomic information. Conclusions Our findings describe a generalizable approach to develop a polygenic score report understandable by potential patients. Although additional studies are needed across a wider spectrum of patient populations, these results are likely to inform ongoing efforts related to polygenic score disclosure within clinical practice.

On the usefulness of CDA-based score reporting: implications for self-regulated learning

In cognitive diagnostic modeling research, one area that has not had enough research interests is remedial learning or instruction based on the information provided by cognitive diagnostic assessment (CDA). The present study tries to address this research gap by looking into the usefulness of the fine-grained score reports based on CDA in two different ways, i.e., a post-test and a survey inquiring about the perceived effectiveness of the score report that provided the skill profile of individual students. Another significance of the current research is that it attempted to introduce cognitive diagnostic assessment into a regular school exam unlike most previous studies that retrofitted to the existing tests. College students in Korea participated in the study, who were encouraged to do self-regulated learning utilizing the detailed information in the CDA-based score report. The results of the post-test and the survey were positive overall, supporting the utility of CDA-generated performance reports. The article ends with some suggestions for future research based on the limitations of the study.

Pleiotropic Variability Score: A Genome Interpretation Metric to Quantify Phenomic Associations of Genomic Variants

A complete understanding of phenomic space is critical for elucidating genome-phenome relationships and assessing disease risk from genome sequencing. We developed a new genome interpretation metric called Pleiotropic Variability Score (PVS) to incorporate phenomic variability into the variant interpretation. PVS uses ontologies of human diseases and medical phenotypes, namely human phenotype ontology (HPO) and disease ontology (DO), to compute the similarities of disease and clinical phenotypes associated with a genetic variant based on semantic reasoning algorithms. We tested 78 unique semantic similarity methods and integrated six robust metrics to define the pleiotropy score of SNPs. We computed PVS for 12, 541 SNPs (10, 021 SNPs mapped to DO phenotype and 8, 569 SNPs mapped to HPO phenotypes) using a repertoire of 382 HPO and 317 DO unique phenotype terms compiled from the genotype-phenotype catalog. We validated the utility of PVS by computing pleiotropy using an electronic health record-linked genomic database (BioME, n=11,210) and generated allele-specific pleiotropy. Further, we demonstrate PVS application in personalized medicine using personalized pleiotropy score reports for individuals with genomic data that could potentially aid in variant interpretation. We further developed a software framework to incorporate PVS into VCF files and consolidate pleiotropy assessment as part of genome interpretation pipelines. As the genome-phenome catalogs are growing, PVS will be a useful metric to assess genetic variation to find SNPs with highly pleiotropic effects. Additionally, genome analysts can prioritize variants with varying degrees of pleiotropy for explorative studies to understand the specific roles of SNPs and pleiotropic hubs in mediating novel phenotypes and drug development.

Design and user experience testing of a polygenic score report: a qualitative study of prospective users

Background: Polygenic scores - which quantify inherited risk by integrating information from many common sites of DNA variation - may enable a tailored approach to clinical medicine. However, alongside considerable enthusiasm, we and others have highlighted a lack of systematic approaches for score disclosure. Here, we review the landscape of polygenic score reporting and describe a generalizable approach for development of polygenic score disclosure tools for coronary artery disease. Methods: First, we assembled a working group of clinicians, geneticists, data visualization specialists, and software developers. The group reviewed existing polygenic score reports and then designed a two-page mock polygenic score report for coronary artery disease. We then conducted a qualitative user-experience study with this report and an interview guide focused on comprehension, experience, and attitudes. Interviews were transcribed and thematically analyzed for themes identification. Results: We conducted interviews with ten adult individuals (50% females, 70% without prior genetic testing experience, age range 20 to 70 years) recruited via an online platform. We identified three themes from interviews: (1) visual elements, such as color and simple graphics, enable participants to interpret, relate to, and contextualize their polygenic score, (2) word-based descriptions of risk and polygenic scores presented as percentiles were most often recognized and understood, (3) participants had varying levels of interest in understanding complex genomic information and therefore would benefit from additional resources that can adapt to their individual needs in real time. In response to user feedback, colors used for communicating risk were modified to minimize unintended color associations and odds ratios were removed. Of note, all 10 participants expressed interest in receiving this report based on their personal genomic information. Conclusions: Our findings describe a generalizable approach to develop and test a polygenic score disclosure tool that is desired by the general public. These results are likely to inform ongoing efforts related to polygenic score disclosure within clinical practice.

State Assessment Score Reporting Practices for English Learner Parents

This study sought to investigate how states communicate results for academic achievement and English language proficiency (ELP) assessments to parents who are English learners (EL). This objective was addressed by evaluating: (a) whether score reports and interpretive guides for state academic achievement and ELP assessments in each state were translated for EL parents; and (b) if so, whether recommended score reporting guidelines were followed in practice. Results demonstrated that for state achievement tests, 29 states had translated score reports and 28 had translated interpretive guides. Nearly every state translated these materials for their ELP assessments in a wide variety of languages. Across ELP and state achievement assessments, most states were found to limit statistical jargon, utilize figures/graphics to communicate test results, and include follow-up information for parents. However, states rarely provided personalization, statements on intended score use, a student’s score history, or a direct link to their interpretive guide in their score reports. Improvements in a number of areas were observed from prior reviews. Recommendations on making score reports and interpretive guides more accessible and interpretable for EL parents are discussed.

Geriatric Fast Facts: Four Years of Growth and Reach

Clinical teachers are increasingly challenged to find the time to provide point of care education for learners. To meet this challenge, an interprofessional team from competing health care systems created and sustained Geriatric Fast Facts (GFF): easily accessible, concise 1-2 page topic summaries for clinical teachers to use (in lieu of the mini-lecture) with learners at the point of care. Designed by geriatrics educators in consultation with IT experts, we launched a mobile enabled website that is indexed and searchable by free text or topic, organ system, ACGME competency, disease, and the “underlying science” for the disease/illness. GFF topics are authored by subject matter experts with peer review by senior geriatricians. Brief quizzes test learners’ knowledge with score reports. Our 4-year results reveal that: 1) “Geriatric Fast Facts” appear in the top Google 10 listing; 2) Search engines account for 39% of site traffic; 3) 60% of unique users (N=29,000) are via direct link; 4) 19% via referral from another site. Our “bounce rate” (55%) is ideal as users quickly gain the information sought - affirmed by our session duration which averages < 2 minutes. Our Google Analytics results reflect steady growth and international reach: 73% of sessions are from the U.S., 5% from Canada, 22% international and growing! In summary with our population continuing to age and time for teaching being limited, GFFs are quick, accessible, evidence-based resources for point of care teaching.

Characteristics and Outcomes of Individuals Engaging in USMLE Irregular Behavior, 2006–2015

ABSTRACT Medical licensing authorities and other entities utilize and rely on the United States Medical Licensing Examination (USMLE) as a standardized, valid and reliable tool to assess physicians’ knowledge and skills. As such, engaging in irregular behavior during the USMLE process can have a broad and damaging impact on an individual’s ability to complete the USMLE sequence and subsequently obtain a medical license in the United States. While there are also repercussions for the USMLE program and entities overseeing medical students and physicians, the risk to the public of being cared for by someone who did not pass a medical licensing examination by his or her own merit is of great concern. This study reviews data about individuals who engaged in irregular behavior, common sanctions taken against them and their ability to ultimately practice medicine in the United States. Using data from the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME), individuals with findings of irregular behavior as part of the USMLE application and/or testing process between 2006 to 2015 were identified, along with their corresponding demographic, examination, sanction and licensure outcome data. A total of 165 individuals were found to have engaged in 170 incidents of irregular behavior. The majority of individuals were male (63%, n = 101), international medical graduates (69%, n = 112) and had a mean age of 33 years old (SD = 8 years, n = 161) at the first incident or finding of irregular behavior. The two most common types of irregular behavior were falsified information (31%, n = 53) and security violations (27%, n = 46). Most incidents (86%, n = 146) were reported to the FSMB Physician Data Center (PDC) and 68% (n = 116) involved a bar from taking the USMLE for a period of time. Only 26% (n = 43) of individuals ultimately passed the entire USMLE sequence and 16% (n = 27) obtained a full, unrestricted medical license in the U.S. by 2019. To help maintain the integrity of a key component for initial licensure in the United States, there is a continued need for rigorous enforcement and safeguarding of USMLE examination applications, content, testing conditions and score reports. Individuals who plan on taking the USMLE should become familiar with USMLE rules and penalties regarding irregular behavior, including the serious implications of such behavior that can severely diminish their ability to practice medicine in the United States. Entities using USMLE information for licensure or admission into medical-related programs need to diligently ensure authentication of USMLE documents and carefully consider if individuals who have engaged in irregular behavior are qualified to practice medicine.

Immediate Post-concussion and Cognitive Testing: Ceiling Effects, Reliability, and Implications for Interpretation

Objective The most commonly used computerized neurocognitive test in concussion assessment and management, Immediate Post-concussion and Cognitive Testing (ImPACT), has demonstrated varying and sometimes concerning levels of test–retest reliability. This study aimed to further examine ImPACT’s psychometric qualities and whether ceiling effects may suppress its reliability. Method A total of 300 consecutively selected ImPACT score reports for students attending a secondary school between 2010 and 2015 were reviewed. Test–retest reliabilities for composite scores and subscales were computed using Pearson product moment correlations (r) and intraclass correlation coefficients. To examine the potential influence of ceiling effects, we conducted frequency analyses of scores falling at, or near, the maximum possible score. Results A total of 92 score reports met inclusion criteria. Test–retest reliabilities ranged from 0.42 to 0.69 for composite scores and 0.19 to 0.71 for subscales. Subscales comprising the Verbal and Visual Memory composites evidenced the most prominent ceiling effects. Conclusions Obtained test–retest reliabilities were consistent with a large segment of the literature and add to concerns about ImPACT’s reliability. Furthermore, at least in a select sample, this study identified sizeable ceiling effects that likely diminish the reliability of some composite scores. To mitigate the risk of false-negative errors on post-injury testing, ImPACT users should be cognizant of such ceiling effects. Supplemental, or alternative, approaches to protect against underestimating baseline functioning also warrant consideration.