Combining Insights of Medical Readability Tools and Machine Learning for Reader-oriented Health Resource Evaluation (Preprint)

2020 ◽  
Author(s):  
Yanmeng Liu ◽  
Meng Ji ◽  
Shanshan Lin ◽  
Mengdan Zhao ◽  
Ziqing Lyu
Keyword(s):  
Machine Learning ◽  
Health Information ◽  
Reading Ability ◽  
Assessment Tool ◽  
Assessment Tools ◽  
Superior Performance ◽  
Online Health Information ◽  
Diverse Backgrounds ◽  
Readability Assessment ◽  
Readability Formulas

BACKGROUND Medical texts on the websites are rich resources for the general public to access health information and get advice to assist them with their health concerns. However, the reading comprehension required for this type of information is far more complex than just reading the text alone, because it often requires a high health knowledge or health literacy in the domain-specific disease area. Furthermore, the reading ability of an individual is also influenced by others factors such as literacy, age, morbidities, social-economic status, interest in a specific health topic, cultural and linguistic background. Literature suggests that traditional readability formulas were designed to give one score for all readers. This inevitably urges for a more adaptive readability assessment tools to evaluate online medical information for people with various backgrounds in a much more comprehensive way. OBJECTIVE The aim of this study was to clarify the existing controversy around the inconsistency among readability formulas, and to build a reader-oriented readability assessment tool, which could automatically estimate the readability of online health information in considering the diverse backgrounds from readers. METHODS The aim of this study was to clarify the existing controversy around the inconsistency among readability formulas, and to build a reader-oriented readability assessment tool, which could automatically estimate the readability of online health information in considering the diverse backgrounds from readers. RESULTS We found that the machine learning readability models integrating multiple readability formulas were more effective to estimate readability of online infectious disease information than the individual readability formula alone. The integrated machine-learning models incorporated the features from the readability formulas, while considered specific backgrounds of readers, which resulted in a more superior performance in the readability classification. CONCLUSIONS The empirical study combined with the existing readability formulas and the machine-learning techniques resulted in more accurate prediction of reading difficulties extended beyond the linguistic features originated from the readability formulas. The proposed assessment tool provides a reader-oriented assessment to be more effective in proxy the health information readability. The key significance of the study includes its reader centeredness, which incorporated the diverse backgrounds from the readers, and its clarification of the relative effectiveness and compatibility of different medical readability tools via machine learning.

Reliability and Evaluation of Health Information Online

2011 ◽  
pp. 181-196
Author(s):  
Elmer V. Bernstam ◽  
Funda Meric-Bernstam
Keyword(s):  
Quality Assessment ◽  
Health Information ◽  
Search Engine ◽  
General Public ◽  
Web Search ◽  
Healthcare Professionals ◽  
General Purpose ◽  
Assessment Tools ◽  
Online Health Information ◽  
Web Search Engine

This chapter discusses the problem of how to evaluate online health information. The quality and accuracy of online health information is an area of increasing concern for healthcare professionals and the general public. We define relevant concepts including quality, accuracy, utility, and popularity. Most users access online health information via general-purpose search engines, therefore we briefly review Web search-engine fundamentals. We discuss desirable characteristics for quality-assessment tools and the available evidence regarding their effectiveness and usability. We conclude with advice for healthcare consumers as they search for health information online.

scholarly journals Developing a Quality Benchmark for Determining the Credibility of Web Health Information- a Protocol of a Gold Standard Approach

2021 ◽  
Vol 3 ◽  
Author(s):  
Lubna Daraz ◽  
Sheila Bouseh
Keyword(s):  
Focus Groups ◽  
Health Information ◽  
Systematic Reviews ◽  
Gold Standard ◽  
Health Knowledge ◽  
Assessment Tools ◽  
Online Health Information ◽  
The Public ◽  
Online Platforms

Background: The current pandemic of COVID-19 has changed the way health information is distributed through online platforms. These platforms have played a significant role in informing patients and the public with knowledge that has changed the virtual world forever. Simultaneously, there are growing concerns that much of the information is not credible, impacting patient health outcomes, causing human lives, and tremendous resource waste. With the increasing use of online platforms, patients/the public require new learning models and sharing medical knowledge. They need to be empowered with strategies to navigate disinformation on online platforms.Methods and Design: To meet the urgent need to combat health “misinformation,” the research team proposes a structured approach to develop a quality benchmark, an evidence-based tool that identifies and addresses the determinants of online health information reliability. The specific methods to develop the intervention are the following: (1) systematic reviews: two comprehensive systematic reviews to understand the current state of the quality of online health information and to identify research gaps, (2) content analysis: develop a conceptual framework based on established and complementary knowledge translation approaches for analyzing the existing quality assessment tools and draft a unique set of quality of domains, (3) focus groups: multiple focus groups with diverse patients/the public and health information providers to test the acceptability and usability of the quality domains, (4) development and evaluation: a unique set of determinants of reliability will be finalized along with a preferred scoring classification. These items will be used to develop and validate a quality benchmark to assess the quality of online health information.Expected Outcomes: This multi-phase project informed by theory will lead to new knowledge that is intended to inform the development of a patient-friendly quality benchmark. This benchmark will inform best practices and policies in disseminating reliable web health information, thus reducing disparities in access to health knowledge and combat misinformation online. In addition, we envision the final product can be used as a gold standard for developing similar interventions for specific groups of patients or populations.

scholarly journals Predicting Health Information Suitability for Children Using Machine-Learning Assisted Selection of Semantic Features (Preprint)

2021 ◽  
Author(s):  
Wenxiu Xie ◽  
Meng Ji ◽  
Yanmeng Liu ◽  
Tianyong Hao ◽  
Chi-Yin Chow
Keyword(s):  
Machine Learning ◽  
Health Literacy ◽  
Health Information ◽  
Learning Algorithms ◽  
Machine Learning Algorithms ◽  
Online Health Information ◽  
Semantic Features ◽  
Young Readers ◽  
Selection Of ◽  
Automatic Feature Selection

BACKGROUND Suitability of health resources for specific readerships represents a critical yet underexplored area of research in health informatics, despite its importance in health literacy and health education. High relevance of health information can improve the suitability and readability of online health educational resources for young readers. It has an important role in developing the health literacy of children with increasing exposure to online health information. Existing research on health resource evaluation is limited to the analysis of the morphological and syntactic complexity. Besides, empirical instruments do not exist to evaluate the suitability of online health information for children. OBJECTIVE We aimed to develop algorithms to predict suitability of online health information for this understudied user group, using a small number of semantic features to provide accurate and convenient tools for automatic prediction of the suitability of online health information for children. METHODS Combining machine learning and linguistic insights, we identified semantic features to predict the suitability of online health information for children, as an emerging and large readership on online health information. The selection of natural language features as predicator variables of algorithms went through initial automatic feature selection using Ridge Classifier, support vector machine, extreme gradient boost, followed by revision by linguists, education experts based on effective health information design. We compared algorithms using the automatically selected features (19) and linguistically enhanced features (20), using the initial features (115) as the baseline. RESULTS Using 5-fold cross-validation, comparing with the baseline (115 features), the Gaussian Naive Bayes model (20 features) achieved statistically higher mean sensitivity (P =0.0206, 95% CI: -0.016, 0.1929); mean specificity (P = 0.0205, 95% CI: -0.016, 0.199); mean AUC (P =0.017, 95% CI: -0.007, 0.140); mean Macro F1 (P =0.0061, 95% CI: 0.016, 0.167). The statistically improved performance of the final model (20 features) stands in contrast with the statistically insignificant changes between the original feature set (115) and the automatically selected features (19): mean sensitivity (P =0.134, 95% CI: -0.1699, 0.0681), mean specificity (P = 0.1001, 95% CI: -0.1389, 0.4017); mean AUC (P =0.0082, 95% CI: 0.0059, 0.1126), and mean macro F1 (P = 0.9796, 95% CI: -0.0555, 0.0548). This demonstrates the importance and effectiveness of combing automatic feature selection and expert-based linguistic revision to develop most effective machine learning algorithms from high-dimensional datasets. CONCLUSIONS Our study developed machine learning algorithms for evaluating health information suitability for children, an important readership who is having increasing reliance on online health information for developing their health literacy. User-adaptive automatic assessment of online health contents holds much promise for distant and remote health education among young readers. Our study leveraged the precision, adaptability of machine learning algorithms and insights from health linguistics to help advance this significant yet understudied area of research.

scholarly journals Applying Computerized-Scoring Models of Written Biological Explanations across Courses and Colleges: Prospects and Limitations

2011 ◽  
Vol 10 (4) ◽  
pp. 379-393 ◽  
Author(s):  
Minsu Ha ◽  
Ross H. Nehm ◽  
Mark Urban-Lurain ◽  
John E. Merrill
Keyword(s):  
Machine Learning ◽  
Assessment Tool ◽  
Biology Education ◽  
Evolutionary Change ◽  
Assessment Tools ◽  
Perfect Agreement ◽  
Student Responses ◽  
Learning Software ◽  
Written Explanations ◽  
Kappa Agreement

Our study explored the prospects and limitations of using machine-learning software to score introductory biology students’ written explanations of evolutionary change. We investigated three research questions: 1) Do scoring models built using student responses at one university function effectively at another university? 2) How many human-scored student responses are needed to build scoring models suitable for cross-institutional application? 3) What factors limit computer-scoring efficacy, and how can these factors be mitigated? To answer these questions, two biology experts scored a corpus of 2556 short-answer explanations (from biology majors and nonmajors) at two universities for the presence or absence of five key concepts of evolution. Human- and computer-generated scores were compared using kappa agreement statistics. We found that machine-learning software was capable in most cases of accurately evaluating the degree of scientific sophistication in undergraduate majors’ and nonmajors’ written explanations of evolutionary change. In cases in which the software did not perform at the benchmark of “near-perfect” agreement (kappa > 0.80), we located the causes of poor performance and identified a series of strategies for their mitigation. Machine-learning software holds promise as an assessment tool for use in undergraduate biology education, but like most assessment tools, it is also characterized by limitations.

scholarly journals Evaluation of Shock Detection Algorithm for Road Vehicle Vibration Analysis

Vibration ◽  
2018 ◽  
Vol 1 (2) ◽  
pp. 220-238 ◽  
Author(s):  
Julien Lepine ◽  
Vincent Rouillard
Keyword(s):  
Machine Learning ◽  
Assessment Tool ◽  
High Amplitude ◽  
Detection Algorithm ◽  
Machine Learning Algorithms ◽  
Assessment Tools ◽  
Detection Methods ◽  
Support Vector ◽  
Road Vehicle ◽  
Shock Detection

The ability to characterize shocks which occur during road transport is a vital prerequisite for the design of optimized protective packaging, which can assist in reducing cost and waste related to products and good transport. Many methods have been developed to detect shocks buried in road vehicle vibration signals, but none has yet considered the nonstationary nature of vehicle vibration and how, individually, they fail to accurately detect shocks. Using machine learning, several shock detection methods can be combined, and the reliability and accuracy of shock detection can also be improved. This paper presents how these methods can be integrated into four different machine learning algorithms (Decision Tree, k-Nearest Neighbors, Bagged Ensemble, and Support Vector Machine). The Pseudo-Energy Ratio/Fall-Out (PERFO) curve, a novel classification assessment tool, is also introduced to calibrate the algorithms and compare their detection performance. In the context of shock detection, the PERFO curve has an advantage over classical assessment tools, such as the Receiver Operating Characteristic (ROC) curve, as it gives more importance to high-amplitude shocks.

scholarly journals Machine Learning   to Assess Language Impairments in Older Adults

2020 ◽  
Author(s):  
Mah Parsa ◽  
Muhammad Raisul Alam ◽  
Alex Mihailidis
Keyword(s):  
Machine Learning ◽  
Older Adults ◽  
Language Impairment ◽  
Assessment Tool ◽  
Language Impairments ◽  
Assessment Tools ◽  
Acoustic Features ◽  
Linguistic Features ◽  
Linguistic Markers ◽  
The Impact

Abstract Objectives: The main objective of this paper is to propose a methodology based on machine learning classifiers for assessing language impairments associated with dementia in older adults. To do so, we compare the impact of different types of language tasks, features, and recording media on our ML-based methodology’s efficiency. Methodology: The methodology encompasses the following steps: 1) Extracting linguistic and acoustic features from subjects’ speeches which have been collected from subjects with dementia ( N =9) and subjects without dementia ( N =13); 2) Employing feature selection methods to rank informative features; 3) Training ML classifiers using extracted features to recognize subjects with dementia from subjects without dementia; 4) Evaluating the classifiers; 5) Selecting the most accurate classifiers to develop the languages assessment tools. Results: Our results indicate that 1) we can find more predictive linguistic markers to distinguish language impairment associated with dementia from participants’ speech produced during the picture description language task than the story recall task. 2) a phone-based recording interface provides a more high-quality language dataset than the web-based recording systems; 3) classifiers trained with selected features from acoustic features or linguistic features show higher performance than the classifiers trained with pure features. Conclusion: Our results show that the tree-based classifiers that have been trained using the PD dataset can be used to develop an ML-based language assessment tool that can detect language impairment associated with dementia as quickly as possible.

Reliability and Evaluation of Health Information Online

2011 ◽  
pp. 2029-2041
Author(s):  
Elmer V. Bernstam ◽  
Funda Meric-Bernstam
Keyword(s):  
Quality Assessment ◽  
Health Information ◽  
Search Engines ◽  
General Public ◽  
Healthcare Professionals ◽  
General Purpose ◽  
Assessment Tools ◽  
Online Health Information

This chapter discusses the problem of how to evaluate online health information. The quality and accuracy of online health information is an area of increasing concern for healthcare professionals and the general public. We define relevant concepts including quality, accuracy, utility, and popularity. Most users access online health information via general-purpose search engines, therefore we briefly review Web searchengine fundamentals. We discuss desirable characteristics for quality-assessment tools and the available evidence regarding their effectiveness and usability. We conclude with advice for healthcare consumers as they search for health information online.

scholarly journals Using the crowd for readability prediction

2012 ◽  
Vol 20 (3) ◽  
pp. 293-325 ◽  
Author(s):  
ORPHÉE DE CLERCQ ◽  
VÉRONIQUE HOSTE ◽  
BART DESMET ◽  
PHILIP VAN OOSTEN ◽  
MARTINE DE COCK ◽  
...  
Keyword(s):  
Machine Learning ◽  
Language Processing ◽  
Assessment Tools ◽  
Good News ◽  
Data Sets ◽  
Readability Score ◽  
Learning Tasks ◽  
Machine Learning Approach ◽  
Knowledge And Attitude ◽  
Readability Assessment

AbstractWhile human annotation is crucial for many natural language processing tasks, it is often very expensive and time-consuming. Inspired by previous work on crowdsourcing, we investigate the viability of using non-expert labels instead of gold standard annotations from experts for a machine learning approach to automatic readability prediction. In order to do so, we evaluate two different methodologies to assess the readability of a wide variety of text material: A more traditional setup in which expert readers make readability judgments and a crowdsourcing setup for users who are not necessarily experts. To this purpose two assessment tools were implemented: a tool where expert readers can rank a batch of texts based on readability, and a lightweight crowdsourcing tool, which invites users to provide pairwise comparisons. To validate this approach, readability assessments for a corpus of written Dutch generic texts were gathered. By collecting multiple assessments per text, we explicitly wanted to level out readers' background knowledge and attitude. Our findings show that the assessments collected through both methodologies are highly consistent and that crowdsourcing is a viable alternative to expert labeling. This is a good news as crowdsourcing is more lightweight to use and can have access to a much wider audience of potential annotators. By performing a set of basic machine learning experiments using a feature set that mainly encodes basic lexical and morpho-syntactic information, we further illustrate how the collected data can be used to perform text comparisons or to assign an absolute readability score to an individual text. We do not focus on optimising the algorithms to achieve the best possible results for the learning tasks, but carry them out to illustrate the various possibilities of our data sets. The results on different data sets, however, show that our system outperforms the readability formulas and a baseline language modelling approach. We conclude that readability assessment by comparing texts is a polyvalent methodology, which can be adapted to specific domains and target audiences if required.

Sign in / Sign up

Export Citation Format

Share Document