scholarly journals Methodology for a statistically sound evaluation of clinical NLP systems (Preprint)

2020 ◽  
Author(s):  
Lea Canales ◽  
Ariel D’Agostino ◽  
Sebastian Menke
Keyword(s):  
Language Processing ◽  
Gold Standard ◽  
Performance Metrics ◽  
Software Tool ◽  
Chronic Obstructive ◽  
Use Case ◽  
Obstructive Pulmonary Disease ◽  
System P ◽  
Primary Variable ◽  
Clinical Natural Language Processing

BACKGROUND Clinical Natural Language Processing (NLP) systems are of crucial importance, because of their increasing relevance in driving decisions about clinical practice. However, carrying out a sound evaluation of NLP systems is complex and hindered by a lack of guidance on how to approach it. OBJECTIVE This research aims to provide a state-of-the-art methodology for the evaluation of a clinical NLP system, thereby guiding NLP researchers in this process with the final goal to ensure the robustness and representativeness of the performance metrics. METHODS We developed a methodology that guides through the process of developing an evaluation of a clinical NLP system using Savana’s ‘EHRead technology’ applied on a real use-case on chronic obstructive pulmonary disease (COPD). In addition, we further introduce SLiCE, a software tool that assists NLP specialists to create a statistically useful gold standard. RESULTS The gold standard contained 49.6% positive and 50.4% negative examples for COPD. For the COPD study, the confidence interval (CI) of the primary variable COPD, calculated using SLiCE, demonstrated its usefulness with CI widths of 0.074 for Precision, 0.046 for Recall, and 0.061 for F1, respectively. CONCLUSIONS Our proposed methodology aims to assist the process of creating an evaluation of a clinical NLP system. Researchers can follow our suggestions step-by-step and use SLiCE to statistically back up their gold standard. We successfully evaluated Savana’s ‘EHRead technology’ using our proposed methodology on a real use-case. We share here the outcome of our experiences working in developing NLP solutions for the clinical domain, hoping that it might help others to establish sound protocols for the evaluation of their NLP system.

scholarly journals Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology

10.2196/20492 ◽  
2021 ◽  
Vol 9 (7) ◽  
pp. e20492
Author(s):  
Lea Canales ◽  
Sebastian Menke ◽  
Stephanie Marchesseau ◽  
Ariel D’Agostino ◽  
Carlos del Rio-Bermudez ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Gold Standard ◽  
Performance Metrics ◽  
Evaluation Methodology ◽  
Free Text ◽  
Use Case ◽  
Five Phases ◽  
Clinical Natural Language Processing

Background Clinical natural language processing (cNLP) systems are of crucial importance due to their increasing capability in extracting clinically important information from free text contained in electronic health records (EHRs). The conversion of a nonstructured representation of a patient’s clinical history into a structured format enables medical doctors to generate clinical knowledge at a level that was not possible before. Finally, the interpretation of the insights gained provided by cNLP systems has a great potential in driving decisions about clinical practice. However, carrying out robust evaluations of those cNLP systems is a complex task that is hindered by a lack of standard guidance on how to systematically approach them. Objective Our objective was to offer natural language processing (NLP) experts a methodology for the evaluation of cNLP systems to assist them in carrying out this task. By following the proposed phases, the robustness and representativeness of the performance metrics of their own cNLP systems can be assured. Methods The proposed evaluation methodology comprised five phases: (1) the definition of the target population, (2) the statistical document collection, (3) the design of the annotation guidelines and annotation project, (4) the external annotations, and (5) the cNLP system performance evaluation. We presented the application of all phases to evaluate the performance of a cNLP system called “EHRead Technology” (developed by Savana, an international medical company), applied in a study on patients with asthma. As part of the evaluation methodology, we introduced the Sample Size Calculator for Evaluations (SLiCE), a software tool that calculates the number of documents needed to achieve a statistically useful and resourceful gold standard. Results The application of the proposed evaluation methodology on a real use-case study of patients with asthma revealed the benefit of the different phases for cNLP system evaluations. By using SLiCE to adjust the number of documents needed, a meaningful and resourceful gold standard was created. In the presented use-case, using as little as 519 EHRs, it was possible to evaluate the performance of the cNLP system and obtain performance metrics for the primary variable within the expected CIs. Conclusions We showed that our evaluation methodology can offer guidance to NLP experts on how to approach the evaluation of their cNLP systems. By following the five phases, NLP experts can assure the robustness of their evaluation and avoid unnecessary investment of human and financial resources. Besides the theoretical guidance, we offer SLiCE as an easy-to-use, open-source Python library.

scholarly journals Underuse of spirometry by general practitioners for the diagnosis of COPD in Italy

2005 ◽  
Vol 63 (1) ◽  
Author(s):  
G. Caramori ◽  
G. Bettoncelli ◽  
R. Tosatto ◽  
F. Arpinelli ◽  
G. Visonà ◽  
...  
Keyword(s):  
Chronic Obstructive Pulmonary Disease ◽  
General Practitioners ◽  
Gold Standard ◽  
Obstructive Lung Disease ◽  
Chronic Obstructive Lung Disease ◽  
Further Education ◽  
Airflow Limitation ◽  
Chronic Obstructive ◽  
Obstructive Pulmonary Disease ◽  
Global Initiative

Background. The Global Initiative for Chronic Obstructive Lung Disease (GOLD) underlines that spirometry is the gold standard as the most reproducible, standardised, and objective way of measuring airflow limitation in the diagnosis and assessment of Chronic Obstructive Pulmonary Disease (COPD). However, studies undertaken in different countries have suggested a widespread underuse of spirometry by general practitioners to establish the diagnosis of COPD. Precise estimates of the prevalence of physician-diagnosed COPD in Italy are not currently available. In collaboration with the Italian Academy of General practitioners (SIMG) we have investigated the degree of use of spirometry to establish the diagnosis of COPD in Italy. Methods. A standardised questionnaire has been selfadministered to a sample of 2425 Italian general practitioners (representing 5% of all the Italian doctors involved in general practice). They have been chosen to cover each of the Italian counties. Results. The prevalence of physician-diagnosed COPD was found to be approximately 4%. However, 30% of general practitioners do not use spirometry to establish the diagnosis of COPD. The main reasons given for the failure to use spirometry are (i) that spirometry is not necessary for the diagnosis of COPD or (ii) there are logistical limitations to the access of the patients to lung function laboratories. Conclusions. This data suggests that contrary to GOLD Guidelines, in Italy, as with other countries, spirometry is not always used in the diagnosis of COPD. There is a clear necessity for further education initiatives targeted to this group of physicians.

scholarly journals Peculiarities of application of pulmonary rehabilitation program in patients with chronic obstructive pulmonary disease

2015 ◽  
Vol 22 (4) ◽  
pp. 34-37 ◽  
Author(s):  
Yu. G. Krylova ◽  
V. I. Trofimov ◽  
A. A. Potapchuk
Keyword(s):  
Social Support ◽  
Chronic Obstructive Pulmonary Disease ◽  
Patient Education ◽  
Pulmonary Disease ◽  
Gold Standard ◽  
Pulmonary Rehabilitation ◽  
Rehabilitation Program ◽  
Chronic Obstructive ◽  
Obstructive Pulmonary Disease ◽  
Pulmonary Rehabilitation Program

Comprehensive pulmonary rehabilitation is gradually becoming the «gold standard» in the treatment of chronic obstructive pulmonary disease. It includes physical training, correction of nutritional status, patient education, psychological and social support.

Therapy-related issues: respiratory system

2017 ◽  
Author(s):  
Philip Wiffen ◽  
Marc Mitchell ◽  
Melanie Snelling ◽  
Nicola Stoner
Keyword(s):  
Chronic Obstructive Pulmonary Disease ◽  
Pulmonary Disease ◽  
Respiratory System ◽  
Asthma Management ◽  
British National Formulary ◽  
Chronic Obstructive ◽  
National Formulary ◽  
Obstructive Pulmonary Disease ◽  
System P ◽  
Current Guidelines

This chapter outlines information relevant to clinical pharmacists related to respiratory system issues and is loosely based on the British National Formulary, Chapter 3. In particular, this chapter covers current guidelines on asthma management, including a section on inhaler techniques for different products, and management of stable chronic obstructive pulmonary disease.

Chest medicine

2014 ◽  
Author(s):  
Murray Longmore ◽  
Ian B. Wilkinson ◽  
Andrew Baldwin ◽  
Elizabeth Wallin
Keyword(s):  
Chronic Obstructive Pulmonary Disease ◽  
Cystic Fibrosis ◽  
Pulmonary Disease ◽  
Respiratory System ◽  
Chronic Obstructive ◽  
Chronic Asthma ◽  
Obstructive Pulmonary Disease ◽  
Lung Tumours ◽  
System P ◽  
Bedside Tests

Signs:Examining the respiratory system –Investigations:Bedside tests in chest medicineFurther investigations in chest medicinePulmonary conditions:PneumoniaSpecific pneumoniasComplications of pneumoniaBronchiectasisCystic fibrosis (cf)Fungi and the lungLung tumoursAsthmaManagement of chronic asthmaChronic obstructive pulmonary disease (...

Sign in / Sign up

Export Citation Format

Share Document