Smart Health System to Detect Dementia Disorders Using Virtual Reality

Smart health technology includes physical sensors, intelligent sensors, and output advice to help monitor patients’ health and adjust their behavior. Virtual reality (VR) plays an increasingly larger role to improve health outcomes, being used in a variety of medical specialties including robotic surgery, diagnosis of some difficult diseases, and virtual reality pain distraction for severe burn patients. Smart VR health technology acts as a decision support system in the diseases diagnostic test of patients as they perform real world tasks in virtual reality (e.g., navigation). In this study, a non-invasive, cognitive computerized test based on 3D virtual environments for detecting the main symptoms of dementia (memory loss, visuospatial defects, and spatial navigation) is proposed. In a recent study, the system was tested on 115 real patients of which thirty had a dementia, sixty-five were cognitively healthy, and twenty had a mild cognitive impairment (MCI). The performance of the VR system was compared with Mini-Cog test, where the latter is used to measure cognitive impaired patients in the traditional diagnosis system at the clinic. It was observed that visuospatial and memory recall scores in both clinical diagnosis and VR system of dementia patients were less than those of MCI patients, and the scores of MCI patients were less than those of the control group. Furthermore, there is a perfect agreement between the standard methods in functional evaluation and navigational ability in our system where P-value in weighted Kappa statistic= 100% and between Mini-Cog-clinical diagnosis vs. VR scores where P-value in weighted Kappa statistic= 93%.

US LI-RADS Visualization Score: Interobserver Variability and Association With Cause of Liver Disease, Sex, and Body Mass Index

Objectives: To evaluate the interobserver agreement between radiologists using the Ultrasound Liver Reporting And Data System (US LI-RADS) visualization score and assess association between visualization score and cause of liver disease, sex, and body mass index (BMI). Methods: This retrospective, single institution, cross-sectional study evaluated 237 consecutive hepatocellular carcinoma surveillance US examinations between March 4, 2017 and September 4, 2017. Five abdominal radiologists independently assigned a US LI-RADS visualization score (A, no or minimal limitations; B, moderate limitations; C, severe limitations). Interobserver agreement was assessed with a weighted Kappa statistic. Association between US visualization score (A vs B or C) and cause of liver disease, sex, and BMI (< or ≥ 25 kg/m2) was evaluated using univariate and multivariate analyses. Results: The average weighted Kappa statistic for all raters was 0.51. A score of either B or C was assigned by the majority of radiologists in 148/237 cases and was significantly associated with cause of liver disease ( P = 0.014) and elevated BMI ( P < 0.001). Subjects with viral liver disease were 3.32 times (95% CI: 1.44-8.38) more likely to have a score of A than those with non-alcoholic steatohepatitis ( P = 0.007). The adjusted odds ratio of visualization score A was 0.249 (95% CI: 0.13-0.48) among those whose BMI was ≥25 kg/m2 vs. BMI < 25 kg/m2. Conclusion: Interobserver agreement between radiologists using US LI-RADS score was moderate. The majority of US examinations were scored as having moderate or severe limitations, and this was significantly associated with non-alcoholic steatohepatitis and increased BMI.

Virtual Versus In-Person Surgical Fellowship Interviews and Ranking Variability: the COVID-19 Experience

Objective: To investigate the effect of interview format changes (in-person to virtual, one-to-one to multiple-to-one) necessitated by the COVID-19 travel restrictions on candidate ranking variabilities.Method: In 2018/2019, the glaucoma fellowship interviews were conducted in-person and one-to-one, whereas in 2020, interviews were virtual and multiple (interviewers)-to-one (candidate). We compared ranking ranges of interviewers within the same virtual room (WSR) and not within the same virtual room (NWSR) to assess the effect of this change on ranking variabilities. We also compared ranking categories (“accept,” “alternate,” and “pass”) agreements between in-person and virtual interviews to assess the effect of this change on ranking variabilities.Results: NWSR and WSR mean rankings differed by 1.33 (95% confidence interval difference 0.61 to 2.04, p = 0.0003), with WSR interviewers having less variability than NWSR pairs. The variability of in-person interviews and later virtual interviews showed no differences (weighted Kappa statistic 0.086 for 2018, 0.158 for 2019, and 0.101 for 2020; p < 0.05 for all years). The overall least attractive candidate has the lowest variability; the most attractive candidate has the second lowest variability.Conclusion: Grouping interviewers decreased ranking variabilities, while a change from in-person to virtual interview format did not increase the ranking variabilities.

The Accuracy Of a Parent-Administered Bleeding Assessment Tool In a Pediatric Hematology Clinic

Background The introduction of bleeding assessment tools (BATs) to quantify the presence and severity of commonly reported bleeding symptoms has received increased interest over the past decade. Bleeding scores, along with laboratory data and family history, can assist the clinician in the assessment of a suspected mild bleeding disorder (MBD). While clinician-administered BATs have been utilized frequently, implementation and validation of the accuracy of a self-report or parent-proxy BAT have yet to be investigated. The primary objective of this study was to determine the accuracy of a parent-administered BAT by measuring the level of agreement between parent and clinician responses to the Condensed MCMDM-1VWD Bleeding Questionnaire. Methods Our study population included children aged 0-19 years presenting to the hematology clinic at Nationwide Children's Hospital (Columbus, OH) for initial evaluation of a suspected MBD or for follow-up evaluation of a previously diagnosed MBD. At the time of the visit, the parent/caregiver completed a short demographic survey and a modified version (targeted for a 6th grade comprehension level) of the Condensed MCMDM-1VWD Bleeding Questionnaire. The treating provider also completed the BAT by interviewing the patient and his/her caregiver; clinicians were blinded to the results of the parent BAT. Both the parent and clinician versions of the BAT were scored and analyzed in the same manner for ease of comparison. We calculated the percentage of agreement and weighted kappa statistic for individual bleeding symptoms as well as the mean across all questionnaire items. We also examined the agreement between caregiver and clinician responses in regards to patient age, gender, diagnosis (new versus follow-up patient), and parent education level. Results To date, we have enrolled 55 eligible patients. The overall mean bleeding score (BS) as calculated from the parent-report BAT was 5.98 (range: -1-25), while the mean BS for the clinician-report BAT was 3.87 (range: 0-16). The mean percentage of agreement between parents and clinicians across all items was 76% (range: 58-98%). The mean weighted kappa statistic was 0.31 (range: -0.04-0.79), representing fair agreement (based on Landis and Koch criteria); the mean Gwet's AC1 (an alternative kappa statistic) was 0.72 (range: 0.48-0.98), representing substantial agreement. Overall, 20% of parent and clinician total bleeding scores matched exactly, and an additional 42% of parent and clinician scores varied by only one to two points. 82% of the study population had an abnormal total bleeding score (defined as ≥2) when rated by parents and 78% had an abnormal total score when rated by clinicians (82% agreement, kappa = 0.43, Gwet's AC1 = 0.73). Tests for equal kappa coefficients did not show significant differences in agreement between parents and clinicians when compared by patient gender, age, diagnosis, or parent education level. Discussion To our knowledge, the results of a patient and/or parent-administered BAT score have not been studied to determine their accuracy and feasibility of use as a screening method for patients with a suspected MBD. While parents tended to over-report bleeding as compared to clinicians, overall, parent and clinician bleeding scores were similar in our study, and these results lend support for the potential use of a modified proxy-report BAT in a clinic setting. Additional research into the construct of the parent-administered BAT is needed to further improve the accuracy of parent-reported bleeding symptoms. Disclosures: Lang: OSUCOM Bennett Medical Student Research Scholarship: Research Funding; ASH HONORS Award: Research Funding.

For Non-expert Clinical Searches, Google Scholar Results are Older with Higher Impact while PubMed Results Offer More Breadth

Objectives – To compare PubMed and Google Scholar results for content relevance and article quality Design – Bibliometric study. Setting – Department of Internal Medicine at Texas Tech University Health Sciences Center. Methods – Four clinical searches were conducted in both PubMed and Google Scholar. Search methods were described as “real world” (p. 216) behaviour, with the searchers familiar with content, though not expert at retrieval techniques. The first 20 results from each search were evaluated for relevance to the initial question, as well as for quality. Relevance was determined based on one author’s subjective assessment of information in the title and abstract, when available, and then tested by two other authors, with discrepancies discussed and resolved. Items were assigned to one of three categories: relevant, possibly relevant, and not relevant to the question, with reviewer agreement measured using a weighted kappa statistic. The quality of items found to be ‘relevant’ and ‘possibly relevant’ was measured by impact factor ratings from Thomsen Reuters (ISI) Web of Knowledge, when available, as well as information obtained by SCOPUS on the number of times items were cited. Main Results – Google Scholar results were judged to be more relevant and of higher quality than results obtained from PubMEed. Google Scholar results are also older on average, while PubMed retrieved items from a larger number of unique journals. Conclusion – In agreement with earlier research, the authors recommended that searchers use both PubMed and Google Scholar to improve on the quality and relevance of results. Searches in the two resources identify unique items based upon the ranking algorithms involved.

Evaluation of Conventional Cytogenetics Versus Fluorescence in Situ Hybridization Analysis in the Prognostic Scoring of Patients with Myelodysplastic Syndromes.

Abstract 1757 Poster Board I-783 Background The myelodysplastic syndromes (MDS) encompass a heterogeneous group of progressive bone marrow insufficiency disorders characterized by chronic cytopenias and a risk of progression to acute leukemias. Non-random clonal chromosomal abnormalities, seen in almost 50% of patients with MDS, are associated with clinically and biologically distinct forms of the disease with prognostic and therapeutic implications. Currently, conventional cytogenetics (CC) and fluorescence in situ hybridization (FISH) techniques are commonly used to detect the chromosomal abnormalities in MDS. In this retrospective study, an effort is made to evaluate the merit of concomitant CC and FISH analyses for prognostic scoring in MDS. Methods A cohort of 86 MDS patients with both CC and FISH results were analyzed. The FISH technique focused on chromosomes 5, 7, 8 and 20. Data was collected on patient age, sex, WHO (World Health Organization) classification, CC and FISH findings, histologic transformation and transfusion dependence. For each patient, IPSS scores were calculated based on either CC (IPSS-CC) or FISH (IPSS-FISH). A weighted kappa statistic was calculated to measure agreement between the CC and FISH methods. Fisher's Exact tests were performed to measure the association between concordance/discordance in scores while survival time for concordant and discordant scores was calculated using Kaplan-Meier estimation and compared using a log-rank test. A significance test of 0.05 was assumed for all comparisons. Results A cohort of 86 MDS patients with 64% male and median age of 68.5 years at diagnosis was studied. According to the WHO classification, 34% of the patients had refractory anemia (RA), 11% had refractory anemia with ringed sideroblasts (RARS), 13% had refractory anemia with excess blasts –1 (RAEB-1), 20% had refractory anemia with excess blasts –2 (RAEB-2) and 20% had refractory cytopenia with multilineage dysplasia (RCMD). Of the 86 patients, 74 had concordant IPSS-CC and IPSS-FISH scores. The p-value for weighted kappa statistic was 0.87 with 95% confidence interval (0.80, 0.94) indicating that there is strong agreement beyond chance between the IPSS-CC and IPSS-FISH. Of the remaining 12 patients with discordant scores, 9 had IPSS-CC > IPSS-FISH while 3 patients (eliminated from comparative analysis for small size) had IPSS-FISH > IPSS-CC. Of the several variables compared between the concordant and discordant groups, only WHO subgroup turned out to be statistically significant. Concordant scores were significantly more likely to be in the RA (37%) and RCMD (23%) cohort, while all discordant patients were in the RAEB-1 (43%), RAEB-2 (29%), and RARS (14%) subgroups (p=0.004). The 9 discordant patients were compared to 27 patients with concordant scores belonging to WHO subgroups RAEB-1, RAEB-2 and RARS. The prognostic variables for comparison were patient survival, leukemic transformation and transfusion dependence. 33% of the discordant patients had leukemic transformation as compared to 11% of concordant patients (p=0.15). Although the difference was large, the sample size was too small to be statistically significant. Conclusion Currently CC and FISH analyses are both being employed to detect the chromosomal abnormalities in MDS patients. This retrospective study indicates that there is statistically significant concordance between cytogenetic and FISH analyses. Conceivably, doing one or other test would most likely be sufficient for prognostic scoring of the vast majority of MDS patients especially those in RA and RCMD subclasses. The discordant scores are significantly more likely to occur in the RAEB-1, RAEB-2 and RARS classes hence justifying this subgroup of patients as potential candidates for both CC and FISH analyses. In our 9 patients with discordant scores (IPSS-CC > IPSS-FISH), a trend was observed towards higher leukemic transformation when compared to WHO subgroup matched concordant patients. While further validation of the current findings by larger prospective studies is awaited, this study unravels the MDS patient sub-population where CC and FISH could be complementary and not superfluous. Disclosures No relevant conflicts of interest to declare.

Measuring Agreement for Ordered Ratings in 3 x 3 Tables

Summary Objectives: When two raters consider a qualitative variable ordered according to three categories, the qualitative agreement is commonly assessed with a symmetrically weighted kappa statistic. However, these statistics can present paradoxes, since they may be insensitive to variations of either complete agreements or disagreements. Methods: Agreement may be summarized by the relative amounts of complete agreements, partial and maximal disagreements beyond chance. Fixing the marginal totals and the trace, we computed symmetrically weighted kappa statistics and we developed a new statistic for qualitative agreements. Data sets from the literature were used to illustrate the methods. Results: We show that agreement may be better assessed with the unweighted kappa index, κc, and a new statistic ζ, which assesses the excess of maximal disagreements with respect to the partial ones, and does not depend on a particular weighting system. When ζis equal to zero, maximal and partial disagreements beyond chance are equal. With its estimated large sample variance, we compared the values of two contingency tables. Conclusions: The (κc, ζ) pair is sensitive to variations in agreements and/or disagreements and enables locating the difference between two qualitative agreements. The qualitative agreement is better with increasing values of κc and ζ.

Cohen’s Weighted Kappa Statistic in Quality Control–Quality Assurance Procedures: Application to Network-Level Contract Pavement Surface Condition Surveys in British Columbia, Canada

Like many provincial and municipal agencies, the British Columbia Ministry of Transportation (BCMoT) contracts out the collection of pavement surface condition data. Because BCMoT is committed to contracts with multiple private contractors, quality assurance (QA) plays a critical role in ensuring that the data are collected accurately and repeatably from year to year. Comprehensive QA testing procedures for surface distress data have been developed and implemented since the data collection has been based on visual ratings with event boards. Control sites that are manually surveyed are used to evaluate whether the contractor is correctly applying the BCMoT pavement surface distress rating system. To date, the QA testing has been based on a composite-index–based criterion for assessing the level of agreement and supplemented with the detailed severity and density rating data. However, the use of a composite index presents some limitations related to the model formulation and weightings assigned to particular distress types. Although the detailed ratings are useful as a diagnostic tool to pinpoint discrepancies, in the disaggregated format, they are not conducive as acceptance criteria for QA testing. Not widely used in the field of engineering, Cohen’s weighted kappa statistic has been applied since the 1960s in other areas to assess the level of agreement beyond chance among raters. The statistic was therefore identified as a possible solution for improving the ministry’s QA surface distress testing process by providing an overall measure of the level of agreement between the detailed manual benchmark survey and the contractor severity and density ratings. The application is described of Cohen’s weighted kappa statistic for visual surface distress survey QA testing using the BCMoT survey and testing procedures as a case study.