Test-Adequacy and Statistical Testing: Combining Different Properties of a Test-Set

2005 ◽  
Author(s):  
S. Kuball ◽  
J. May
Keyword(s):  
Statistical Testing ◽  
Test Set ◽  
Test Adequacy

scholarly journals STATISTICAL TESTING TECHNIQUE FOR COMPARISON MACHINE LEARNING MODELS PERFORMANCE

2019 ◽  
pp. 10-17
Author(s):  
Yu. S. Fedorenko
Keyword(s):  
Machine Learning ◽  
T Test ◽  
Statistical Testing ◽  
Independent Random Variables ◽  
Learning Models ◽  
Testing Technique ◽  
Test Set ◽  
Student’S T ◽  
Machine Learning Models ◽  
Student’S T Test

The statistical testing technique is considered to compare the metrics values of machine learning models on a test set. Since the values of metrics depend not only on the models, but also on the data, it may turn out that different models are the best on different test sets. For this reason, the traditional approach to comparing the values of metrics on a test set is often not enough. Sometimes a statistical comparison of the results obtained on the basis of cross-validation is used, but in this case it is impossible to guarantee the independence of the obtained measurements, which does not allow the use of the Student's t-test. There are criteria that do not require independent measurements, but they have less power. For additive metrics, a technique is proposed in this paper, when a test sample is divided into N parts, on each of which the values of the metrics are calculated. Since the value on each part is obtained as the sum of independent random variables, according to the central limit theorem, the obtained metrics values on each of the N parts are realizations of the normally distributed random variable. To estimate the required sample size, it is proposed to use normality tests and build quantile– quantile plots. You can then use a modification of the Student's t-test to conduct a statistical test comparing the mean values of the metrics. A simplified approach is also considered, in which confidence intervals are built for the base model. A model whose metric values do not fall into this interval works differently from the base model. This approach reduces the amount of computations needed, however, an experimental analysis of the binary cross-entropy metric for CTR (Click-Through Rate) prediction models showed that it is more rough than the first one.

Using Connectionist Modules for Decision Support

1990 ◽  
Vol 29 (03) ◽  
pp. 167-181 ◽  
Author(s):  
G. Hripcsak
Keyword(s):  
Decision Support ◽  
Standard Deviation ◽  
Confidence Interval ◽  
Posterior Probability ◽  
Back Propagation ◽  
Connectionist Model ◽  
Test Set ◽  
The Third ◽  
Independent Test ◽  
Better Than

AbstractA connectionist model for decision support was constructed out of several back-propagation modules. Manifestations serve as input to the model; they may be real-valued, and the confidence in their measurement may be specified. The model produces as its output the posterior probability of disease. The model was trained on 1,000 cases taken from a simulated underlying population with three conditionally independent manifestations. The first manifestation had a linear relationship between value and posterior probability of disease, the second had a stepped relationship, and the third was normally distributed. An independent test set of 30,000 cases showed that the model was better able to estimate the posterior probability of disease (the standard deviation of residuals was 0.046, with a 95% confidence interval of 0.046-0.047) than a model constructed using logistic regression (with a standard deviation of residuals of 0.062, with a 95% confidence interval of 0.062-0.063). The model fitted the normal and stepped manifestations better than the linear one. It accommodated intermediate levels of confidence well.

Hubungan Religiusitas dengan Citra Tubuh pada Wanita Dewasa Awal

2018 ◽  
Vol 1 (1) ◽  
pp. 9-28
Author(s):  
Dessy Sumanty ◽  
Deden Sudirman ◽  
Diah Puspasari
Keyword(s):  
Body Image ◽  
Correlation Coefficient ◽  
Research Design ◽  
Calculation Result ◽  
The Body ◽  
Statistical Testing ◽  
P Value ◽  
Significance Level ◽  
Correlational Research

This research attempts to relate the body image phenomenon with the level of subject religiosity. This research used correlational research design that was involving 332 respondents. The statistical testing which is used to test the hypothesis Rank Spearman. The calculation result with the significance level of trust 95% (a = 0.05) show that the correlation coefficient is 0.083 and p-value is 0.129. It means that Ho is accepted and H1 is rejected. It can be concluded that there is no relationship between religiosity with body image.

RetroBioCat: Computer-Aided Synthesis Planning for Biocatalytic Reactions and Cascades

2020 ◽  
Author(s):  
William Finnigan ◽  
Lorna J. Hepworth ◽  
Nicholas J. Turner ◽  
Sabine Flitsch
Keyword(s):  
Organic Chemistry ◽  
Computer Aided Design ◽  
Selective Synthesis ◽  
Test Set ◽  
Synthesis Design ◽  
Computer Aided ◽  
Synthesis Planning ◽  
Enzymatic Cascades ◽  
Target Molecules ◽  
Aided Design

As the enzyme toolbox for biocatalysis has expanded, so has the potential for the construction of powerful enzymatic cascades for efficient and selective synthesis of target molecules. Additionally, recent advances in computer-aided synthesis planning (CASP) are revolutionizing synthesis design in both synthetic biology and organic chemistry. However, the potential for biocatalysis is not well captured by tools currently available in either field. Here we present RetroBioCat, an intuitive and accessible tool for computer-aided design of biocatalytic cascades, freely available at retrobiocat.com. Our approach uses a set of expertly encoded reaction rules encompassing the enzyme toolbox for biocatalysis, and a system for identifying literature precedent for enzymes with the correct substrate specificity where this is available. Applying these rules for automated biocatalytic retrosynthesis, we show our tool to be capable of identifying promising biocatalytic pathways to target molecules, validated using a test-set of recent cascades described in the literature.

PREDIKSI KUALITAS AIR SUNGAI CILIWUNG DENGAN MENGGUNAKAN ALGORITMA POHON KEPUTUSAN

2021 ◽  
Vol 12 (2) ◽  
Author(s):  
Mohammad Haekal ◽  
Henki Bayu Seta ◽  
Mayanda Mega Santoni
Keyword(s):  
Data Mining ◽  
Decision Tree ◽  
Cross Validation ◽  
Online Monitoring ◽  
Training Set ◽  
Microsoft Excel ◽  
Test Set

Untuk memprediksi kualitas air sungai Ciliwung, telah dilakukan pengolahan data-data hasil pemantauan secara Online Monitoring dengan menggunakan Metode Data Mining. Pada metode ini, pertama-tama data-data hasil pemantauan dibuat dalam bentuk tabel Microsoft Excel, kemudian diolah menjadi bentuk Pohon Keputusan yang disebut Algoritma Pohon Keputusan (Decision Tree) mengunakan aplikasi WEKA. Metode Pohon Keputusan dipilih karena lebih sederhana, mudah dipahami dan mempunyai tingkat akurasi yang sangat tinggi. Jumlah data hasil pemantauan kualitas air sungai Ciliwung yang diolah sebanyak 5.476 data. Hasil klarifikasi dengan Pohon Keputusan, dari 5.476 data ini diperoleh jumlah data yang mengindikasikan sungai Ciliwung Tidak Tercemar sebanyak 1.059 data atau sebesar 19,3242%, dan yang mengindikasikan Tercemar sebanyak 4.417 data atau 80,6758%. Selanjutnya data-data hasil pemantauan ini dievaluasi menggunakan 4 Opsi Tes (Test Option) yaitu dengan Use Training Set, Supplied Test Set, Cross-Validation folds 10, dan Percentage Split 66%. Hasil evaluasi dengan 4 opsi tes yang digunakan ini, semuanya menunjukkan tingkat akurasi yang sangat tinggi, yaitu diatas 99%. Dari data-data hasil peneltian ini dapat diprediksi bahwa sungai Ciliwung terindikasi sebagai sungai tercemar bila mereferensi kepada Peraturan Pemerintah Republik Indonesia nomor 82 tahun 2001 dan diketahui pula bahwa penggunaan aplikasi WEKA dengan Algoritma Pohon Keputusan untuk mengolah data-data hasil pemantauan dengan mengambil tiga parameter (pH, DO dan Nitrat) adalah sangat akuran dan tepat. Kata Kunci : Kualitas air sungai, Data Mining, Algoritma Pohon Keputusan, Aplikasi WEKA.

Sign in / Sign up

Export Citation Format

Share Document