scholarly journals Looking for a reference for large datasets: relative reliability of visual and automatic sleep scoring

2019 ◽  
Author(s):  
C. Berthomier ◽  
V. Muto ◽  
C. Schmidt ◽  
G. Vandewalle ◽  
M. Jaspar ◽  
...  
Keyword(s):  
Large Datasets ◽  
Automated Scoring ◽  
Scoring Method ◽  
Sleep Scoring ◽  
Sleep Studies ◽  
Fine Grain ◽  
Visual Scoring ◽  
Kappa Value ◽  
Consensus Scoring ◽  
Statistical Solutions

AbstractStudy ObjectivesNew challenges in sleep science require to describe fine grain phenomena or to deal with large datasets. Beside the human resource challenge of scoring huge datasets, the inter- and intra-expert variability may also reduce the sensitivity of such studies. Searching for a way to disentangle the variability induced by the scoring method from the actual variability in the data, visual and automatic sleep scorings of healthy individuals were examined.MethodsA first dataset (DS1, 4 recordings) scored by 6 experts plus an autoscoring algorithm was used to characterize inter-scoring variability. A second dataset (DS2, 88 recordings) scored a few weeks later was used to investigate intra-expert variability. Percentage agreements and Conger’s kappa were derived from epoch-by-epoch comparisons on pairwise, consensus and majority scorings.ResultsOn DS1 the number of epochs of agreement decreased when the number of expert increased, in both majority and consensus scoring, where agreement ranged from 86% (pairwise) to 69% (all experts). Adding autoscoring to visual scorings changed the kappa value from 0.81 to 0.79. Agreement between expert consensus and autoscoring was 93%. On DS2 intra-expert variability was evidenced by the kappa systematic decrease between autoscoring and each single expert between datasets (0.75 to 0.70).ConclusionsVisual scoring induces inter- and intra-expert variability, which is difficult to address especially in big data studies. When proven to be reliable and if perfectly reproducible, autoscoring methods can cope with intra-scorer variability making them a sensible option when dealing with large datasets.Statement of SignificanceWe confirmed and extended previous findings highlighting the intra- and inter-expert variability in visual sleep scoring. On large datasets those variability issues cannot be completely addressed by neither practical nor statistical solutions such as group training, majority or consensus scoring.When an automated scoring method can be proven to be as reasonably imperfect as visual scoring but perfectly reproducible, it can serve as a reliable scoring reference for sleep studies.

scholarly journals Assessment of Ki67 in Breast Cancer: Updated Recommendations from the International Ki67 in Breast Cancer Working Group

2020 ◽  
Author(s):  
Torsten O Nielsen ◽  
Samuel C Y Leung ◽  
David L Rimm ◽  
Andrew Dodson ◽  
Balazs Acs ◽  
...  
Keyword(s):  
Breast Cancer ◽  
Working Group ◽  
Clinical Utility ◽  
Current Evidence ◽  
Automated Scoring ◽  
Scoring Method ◽  
Control Programs ◽  
Visual Scoring ◽  
Er Positive ◽  
Further Development

Abstract Ki67 immunohistochemistry, commonly used as a proliferation marker in breast cancer, has limited value for treatment decisions due to questionable analytical validity. The International Ki67 Working Group (IKWG) consensus meeting, held in October 2019, assessed the current evidence for Ki67 immunohistochemistry analytical validity and clinical utility in breast cancer, including the series of scoring studies the IKWG conducted on centrally stained tissues. Consensus observations and recommendations are: 1) as for estrogen receptor and HER2 testing, pre-analytical handling considerations are critical. 2) A standardized visual scoring method has been established and is recommended for adoption. 3) Participation in and evaluation of quality assurance and quality control programs is recommended to maintain analytical validity. 4) The IKWG accepted that Ki67 IHC as a prognostic marker in breast cancer has clinical validity but concluded that clinical utility is evident only for prognosis estimation in anatomically favorable ER-positive and HER2-negative patients, to identify those who do not need adjuvant chemotherapy. In this T1-2, N0-1 patient group, the IKWG consensus is that Ki67 ≤ 5% or ≥ 30% can be used to estimate prognosis. In conclusion, analytical validity of Ki67 IHC can be reached with careful attention to pre-analytical issues and calibrated standardized visual scoring. Currently, clinical utility of Ki67 IHC in breast cancer care remains limited to prognosis assessment in stage I/II breast cancer. Further development of automated scoring might help to overcome some current limitations.

scholarly journals Digital imaging outperforms traditional scoring methods of spittlebug tolerance in Urochloa humidicola hybrids

2020 ◽  
Author(s):  
Luis M. Hernandez ◽  
Paula Espitia ◽  
Valheria Castiblanco ◽  
Juan A Cardoso
Keyword(s):  
Correlation Coefficient ◽  
Digital Imaging ◽  
Digital Images ◽  
Breeding Program ◽  
Alternative Methods ◽  
Screening Methods ◽  
Plant Tolerance ◽  
Scoring Method ◽  
Plant Damage ◽  
Visual Scoring

AbstractAmerican spittlebug complex (Hemiptera: Cercopidae) is a critical pest for existing Urochloa humidicola cultivars in the neotropical savannas. The U. humidicola breeding program of the International Center for Tropical Agriculture aims to increase tolerance to spittlebugs. To develop U. humidicola genotypes with superior tolerance to spittlebugs than existing cultivars, adequate screening methods ought to be deployed. Currently, visual scores of plant damage by spittlebugs is the standard method to screen for variation in plant tolerance. However, visual scoring is prone to human bias, is of medium throughput and relies of the expertise of well-trained personnel. In this study, we compared estimations of plant damage from two alternative methods (SPAD measurements and digital images) and visual scoring from an inexpert evaluator with the plant damage estimated from an expert. This information should instruct if different methods could be implemented in the U. humidicola breeding program. Time needed to evaluate damage was recorded for each method. Lin’s correlation coefficient, Pearson’s correlation coefficient and broad sense heritability values were also calculated. Overall, damage estimated from digital images showed the highest throughput (twice as fast as visual scoring from an expert); high correlations with visual scoring (r > 0.80, p < 0.0001); and heritability values for plant damage as good or better (> 0.7) than those obtained by visual scoring from an expert. Our results indicate that digital imaging is a phenotyping method that might improve the efficiency of breeding for increased tolerance to spittlebugs in U. humidicola.HighlightDigital imaging outperformed standard scoring method of spittlebug tolerance in Urochloa humidicola, suggesting that this method might improve the efficiency of breeding for such stress.

scholarly journals Automated scoring method for the CDMAM phantom

2009 ◽  
Author(s):  
M. Yip ◽  
W. Chukwu ◽  
E. Kottis ◽  
E. Lewis ◽  
J. Oduko ◽  
...  
Keyword(s):  
Automated Scoring ◽  
Scoring Method

SU-GG-I-122: An Automated Scoring Method to Quantitatively Evaluate Imaging Artifacts in 4D-CT Images

2010 ◽  
Vol 37 (6Part4) ◽  
pp. 3129-3129
Author(s):  
G Cui ◽  
B Jew ◽  
T Yamamoto ◽  
J Hong ◽  
B Loo ◽  
...  
Keyword(s):  
Ct Images ◽  
Automated Scoring ◽  
Scoring Method ◽  
4D Ct ◽  
Imaging Artifacts

Can a digital slide scanner and viewing technique assist the visual scoring for the cytokinesis-block micronucleus cytome assay?

Mutagenesis ◽  
2020 ◽  
Vol 35 (4) ◽  
pp. 311-318
Author(s):  
Emma L Jaunay ◽  
Varinderpal S Dhillon ◽  
Susan J Semple ◽  
Bradley S Simpson ◽  
Permal Deo ◽  
...  
Keyword(s):  
Dna Damage ◽  
Digital Pathology ◽  
Scoring Methods ◽  
High Definition ◽  
Digital Slide ◽  
Scoring Method ◽  
Desktop Computer ◽  
Damage Indices ◽  
Visual Scoring ◽  
Slide Scanner

Abstract The cytokinesis-block micronucleus cytome (CBMNcyt) assay is a comprehensive method to measure DNA damage, cytostasis and cytotoxicity caused by nutritional, radiation and chemical factors. A slide imaging technique has been identified as a new method to assist with the visual scoring of cells for the CBMNcyt assay. A NanoZoomer S60 Digital Pathology slide scanner was used to view WIL2-NS cells treated with hydrogen peroxide (H2O2) and measure CBMNcyt assay biomarkers using a high-definition desktop computer screen. The H2O2-treated WIL2-NS cells were also scored visually using a standard light microscope, and the two visual scoring methods were compared. Good agreement was found between the scoring methods for all DNA damage indices (micronuclei, nucleoplasmic bridges and nuclear buds) and nuclear division index with correlation R values ranging from 0.438 to 0.789, P &lt; 0.05. Apoptotic and necrotic cell frequency was lower for the NanoZoomer scoring method, but necrotic frequency correlated well with the direct visual microscope method (R = 0.703, P &lt; 0.0001). Considerable advantages of the NanoZoomer scoring method compared to direct visual microscopy includes reduced scoring time, improved ergonomics and a reduction in scorer fatigue. This study indicates that a digital slide scanning and viewing technique may assist with visual scoring for the CBMNcyt assay and provides similar results to conventional direct visual scoring.

scholarly journals Signal Amplification of FISH for Automated Detection Using Image Cytometry

1997 ◽  
Vol 13 (3) ◽  
pp. 137-146 ◽  
Author(s):  
Khuong Truong ◽  
Jack Boenders ◽  
Zofia Maciorowski ◽  
Philippe Vielh ◽  
Bernard Dutrillaux ◽  
...  
Keyword(s):  
Signal Amplification ◽  
Human Lymphocytes ◽  
Ccd Camera ◽  
Image Cytometry ◽  
Automated Detection ◽  
Image Analyzer ◽  
Automated Scoring ◽  
Visual Scoring ◽  
Signal Characteristics ◽  
Amplification System

The purpose of this study was to improve the detection of FISH signals, in order that spot counting by a fully automated image cytometer be comparable to that obtained visually under the microscope. Two systems of spot scoring, visual and automated counting, were investigated in parallel on stimulated human lymphocytes with FISH using a biotinylated centromeric probe for chromosome 3. Signal characteristics were first analyzed on images recorded with a coupled charge device (CCD) camera. Number of spots per nucleus were scored visually on these recorded images versus automatically with a DISCOVERY image analyzer. Several fluochromes, amplification systems and pretreatments were tested. Our results for both visual and automated scoring show that the tyramide amplification system (TSA) gives the best amplification of signal if pepsin treatment is applied prior to FISH. Accuracy of the automated scoring, however, remained low (58% of nuclei containing two spots) compared to the visual scoring because of the high intranuclear variation between FISH spots.

Comparison of an Aggregate Scoring Method With a Consensus Scoring Method in a Measure of Clinical Reasoning Capacity

2002 ◽  
Vol 14 (3) ◽  
pp. 150-156 ◽  
Author(s):  
Bernard Charlin ◽  
Martin Desaulniers ◽  
Robert Gagnon ◽  
Daniel Blouin ◽  
Cees van der Vleuten
Keyword(s):  
Clinical Reasoning ◽  
Scoring Method ◽  
Consensus Scoring

scholarly journals POS1401 ASSESSMENT OF INTERREADER RELIABILITY IN SCORING PATIENTS WITH HAND OSTEOARTHRITIS AND PSORIATIC ARTHRITIS BY FLUORESCENCE OPTICAL IMAGING

2021 ◽  
Vol 80 (Suppl 1) ◽  
pp. 983.2-983
Author(s):  
B. Drude ◽  
Ø. Maugesten ◽  
S. G. Werner ◽  
G. R. Burmester ◽  
J. Berger ◽  
...  
Keyword(s):  
Psoriatic Arthritis ◽  
Optical Imaging ◽  
Phase 1 ◽  
Hand Osteoarthritis ◽  
Scoring Method ◽  
Cohen’S Kappa ◽  
Fluorescence Optical Imaging ◽  
Cohen's Kappa ◽  
Kappa Value ◽  
Pip Joint

Background:Fluorescence Optical Imaging (FOI) utilises the fluorophore indocyanine green (ICG) to reflect enhanced microcirculation in hand and finger joints due to inflammation.Objectives:We wanted to assess the interreader reliability of FOI enhancement in patients with hand osteoarthritis (OA) and psoriatic arthritis (PsA). Furthermore, predefined typical morphologic patterns were included to determine the ability of FOI to discriminate between both diagnoses.Methods:An atlas with example images of grade 0-3 in different joint groups and typical morphologic patterns (‘streaky signals’[1], ‘green/blue nail sign’[2], ‘Werner sign’[3,4], and ‘Bishop’s crozier sign’) of PsA and hand OA was created. Two readers scored all joints in both hands (30 in total) of 20 cases with hand OA and PsA. The cases were randomly mixed and both readers were blinded to diagnosis. Each joint was rated on a semiquantitative scale from 0 to 3 in five different images (PrimaVista Mode (PVM), phase 1, 2 (first and middle image), and 3) during the FOI sequence according to the scoring method FOIAS (fluorescence optical imaging activity score)[1,3]. Interreader reliability on scoring joint enhancement was calculated using linear weighted Cohen’s kappa (κ). Agreement on diagnosis (hand OA vs. PsA) and different morphologic patterns was assessed by calculating (regular) Cohen’s kappa.Results:Overall agreement on scoring joint enhancement (all phases) was substantial (κ = 0.75), with greatest consensus in phase 2 first (κ = 0.75) and lowest agreement in phase 1 (κ = 0.46). Reliability varied in different joint groups (wrist, MCP, (P)IP, DIP), with almost perfect overall agreement on PIP joint affection (κ = 0.81), substantial agreement on wrist (κ = 0.69) and DIP joint affection (κ = 0.63), and moderate agreement on MCP joint affection (κ = 0.49) across all phases. Consensus on morphologic patterns showed overall fair agreement (κ = 0.37) with a similar kappa value on the ability to discriminate between both diagnoses (κ = 0.3).Conclusion:Joint enhancement in FOI can be reliably assessed using a predefined scoring method. The ability of FOI to differentiate between hand OA and PsA seems to be limited. Clearer definition and more training might be needed to better agree on morphologic patterns in FOI.References:[1] Glimm AM, Werner SG, Burmester GR, et al. Ann Rheum Dis. 2016 Mar;75(3):566-570[2] Wiemann O, Werner SG, Langer HE, et al. J Dtsch Dermatol Ges. 2019 Feb;17(2):138-148[3] Werner SG, Langer HE, Ohrndorf S, et al. Ann Rheum Dis. 2012 Apr;71(4):504-510[4] Zeidler H 2019. Fluoreszenzoptische Bildgebung. In: Zeidler H, Michel BA. Differenzialdiagnose rheumatischer Erkrankungen 5. Aufl. Springer, Heidelberg, S. 88-89Disclosure of Interests:Benedict Drude: None declared, Øystein Maugesten: None declared, Stephanie Gabriele Werner: None declared, Gerd Rüdiger Burmester: None declared, Jörn Berger Employee of: Xiralite GmbH, Ida K. Haugen: None declared, Sarah Ohrndorf: None declared

Sign in / Sign up

Export Citation Format

Share Document