Thoughts on Model Validation for Engineering Design

2003 ◽  
Author(s):  
George A. Hazelrigg
Keyword(s):  
Predictive Model ◽  
Engineering Design ◽  
Decision Maker ◽  
Model Validation ◽  
Predictive Models ◽  
Specific Model ◽  
System Behavior ◽  
Crucial Element ◽  
Model Validity ◽  
Subjective Input

Models are the basis for all prediction of system behavior, and hence form a crucial element of engineering design. A key concern is the validity of such models. This paper discusses the notion of model validity and the limits of what one can say about the validity of a specific model. It is shown that predictive models, such as those used in engineering design, cannot be validated objectively. That is, the validation of a predictive model can be accomplished only in the context of a specific decision, and only in the context of subjective input from the decision maker, including preferences.

Some Metrics and a Bayesian Procedure for Validating Predictive Models in Engineering Design

2006 ◽  
Author(s):  
Wei Chen ◽  
Ying Xiong ◽  
Kwok-Leung Tsui ◽  
Shuchun Wang
Keyword(s):  
Engineering Design ◽  
Model Validation ◽  
Computer Model ◽  
Bayesian Approach ◽  
Model Prediction ◽  
Predictive Models ◽  
Design Decision ◽  
Validation Metrics ◽  
Model Based ◽  
Physical Experiments

Even though model-based simulations are widely used in engineering design, it remains a challenge to validate models and assess the risks and uncertainties associated with the use of predictive models for design decision making. In most of the existing work, model validation is viewed as verifying the model accuracy, measured by the agreement between computational and experimental results. However, from the design perspective, a good model is considered as the one that can provide the discrimination (good resolution) between design candidates. In this work, a Bayesian approach is presented to assess the uncertainty in model prediction by combining data from both physical experiments and the computer model. Based on the uncertainty quantification of model prediction, some design-oriented model validation metrics are further developed to guide designers for achieving high confidence of using predictive models in making a specific design decision. We demonstrate that the Bayesian approach provides a flexible framework for drawing inferences for predictions in the intended but may be untested design domain, where design settings of physical experiments and the computer model may or may not overlap. The implications of the proposed validation metrics are studied, and their potential roles in a model validation procedure are highlighted.

Study of the route correction system for the on-board navigation system of an unmanned aerial vehicle on the grounds of radar images of the terrain

2020 ◽  
pp. 129-134
Keyword(s):  
Predictive Model ◽  
Unmanned Aerial Vehicle ◽  
Predictive Models ◽  
Navigation System ◽  
Error Compensation ◽  
Compensation Scheme ◽  
Radar Images ◽  
Aerial Vehicle ◽  
System Errors ◽  
Correction System

The system of route correction of an unmanned aerial vehicle (UAV) is considered. For the route correction the on-board radar complex is used. In conditions of active interference, it is impossible to use radar images for the route correction so it is proposed to use the on-board navigation system with algorithmic correction. An error compensation scheme of the navigation system in the output signal using the algorithm for constructing a predictive model of the system errors is applied. The predictive model is building using the genetic algorithm and the method of group accounting of arguments. The quality comparison of the algorithms for constructing predictive models is carried out using mathematical modeling.

O-203 Application of machine learning to predict aneuploidy and mosaicism in embryos from in vitro fertilization (IVF) cycles

2021 ◽  
Vol 36 (Supplement_1) ◽  
Author(s):  
J A Ortiz ◽  
R Morales ◽  
B Lledo ◽  
E Garcia-Hernandez ◽  
A Cascales ◽  
...  
Keyword(s):  
Machine Learning ◽  
Predictive Model ◽  
Predictive Models ◽  
Maternal Age ◽  
The Other ◽  
Predictor Variables ◽  
Learning Models ◽  
Male Factor ◽  
Factors Associated ◽  
Machine Learning Models

Abstract Study question Is it possible to predict the likelihood of an IVF embryo being aneuploid and/or mosaic using a machine learning algorithm? Summary answer There are paternal, maternal, embryonic and IVF-cycle factors that are associated with embryonic chromosomal status that can be used as predictors in machine learning models. What is known already The factors associated with embryonic aneuploidy have been extensively studied. Mostly maternal age and to a lesser extent male factor and ovarian stimulation have been related to the occurrence of chromosomal alterations in the embryo. On the other hand, the main factors that may increase the incidence of embryo mosaicism have not yet been established. The models obtained using classical statistical methods to predict embryonic aneuploidy and mosaicism are not of high reliability. As an alternative to traditional methods, different machine and deep learning algorithms are being used to generate predictive models in different areas of medicine, including human reproduction. Study design, size, duration The study design is observational and retrospective. A total of 4654 embryos from 1558 PGT-A cycles were included (January-2017 to December-2020). The trophoectoderm biopsies on D5, D6 or D7 blastocysts were analysed by NGS. Embryos with ≤25% aneuploid cells were considered euploid, between 25-50% were classified as mosaic and aneuploid with >50%. The variables of the PGT-A were recorded in a database from which predictive models of embryonic aneuploidy and mosaicism were developed. Participants/materials, setting, methods The main indications for PGT-A were advanced maternal age, abnormal sperm FISH and recurrent miscarriage or implantation failure. Embryo analysis were performed using Veriseq-NGS (Illumina). The software used to carry out all the analysis was R (RStudio). The library used to implement the different algorithms was caret. In the machine learning models, 22 predictor variables were introduced, which can be classified into 4 categories: maternal, paternal, embryonic and those specific to the IVF cycle. Main results and the role of chance The different couple, embryo and stimulation cycle variables were recorded in a database (22 predictor variables). Two different predictive models were performed, one for aneuploidy and the other for mosaicism. The predictor variable was of multi-class type since it included the segmental and whole chromosome alteration categories. The dataframe were first preprocessed and the different classes to be predicted were balanced. A 80% of the data were used for training the model and 20% were reserved for further testing. The classification algorithms applied include multinomial regression, neural networks, support vector machines, neighborhood-based methods, classification trees, gradient boosting, ensemble methods, Bayesian and discriminant analysis-based methods. The algorithms were optimized by minimizing the Log_Loss that measures accuracy but penalizing misclassifications. The best predictive models were achieved with the XG-Boost and random forest algorithms. The AUC of the predictive model for aneuploidy was 80.8% (Log_Loss 1.028) and for mosaicism 84.1% (Log_Loss: 0.929). The best predictor variables of the models were maternal age, embryo quality, day of biopsy and whether or not the couple had a history of pregnancies with chromosomopathies. The male factor only played a relevant role in the mosaicism model but not in the aneuploidy model. Limitations, reasons for caution Although the predictive models obtained can be very useful to know the probabilities of achieving euploid embryos in an IVF cycle, increasing the sample size and including additional variables could improve the models and thus increase their predictive capacity. Wider implications of the findings Machine learning can be a very useful tool in reproductive medicine since it can allow the determination of factors associated with embryonic aneuploidies and mosaicism in order to establish a predictive model for both. To identify couples at risk of embryo aneuploidy/mosaicism could benefit them of the use of PGT-A. Trial registration number Not Applicable

scholarly journals The effect of Kurtosis on the accuracy of artificial neural network predictive model

2018 ◽  
Vol 204 ◽  
pp. 02018
Author(s):  
Aisyah Larasati ◽  
Anik Dwiastutik ◽  
Darin Ramadhanti ◽  
Aal Mahardika
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Predictive Model ◽  
Predictive Models ◽  
Output Layer ◽  
Input Layer ◽  
Artificial Neural ◽  
Misclassification Rates ◽  
Hidden Layer ◽  
Program Data

This study aims to explore the effect of kurtosis level of the data in the output layer on the accuracy of artificial neural network predictive models. The artificial neural network predictive models are comprised of one node in the output layer and six nodes in the input layer. The number of hidden layer is automatically built by the program. Data are generated using simulation approach. The results show that the kurtosis level of the node in the output layer is significantly affect the accuracy of the artificial neural network predictive model. Platycurtic and leptocurtic data has significantly higher misclassification rates than mesocurtic data. However, the misclassification rates between platycurtic and leptocurtic is not significantly different. Thus, data distribution with kurtosis nearly to zero results in a better ANN predictive model.

TASKILLAN: A Simulation to Predict the Validity of Multiple Resource Models of Aviation Workload

1988 ◽  
Vol 32 (2) ◽  
pp. 168-172 ◽  
Author(s):  
Christopher D. Wickens ◽  
Kelly Harwood ◽  
Leon Segal ◽  
Inge Tkalcevic ◽  
Bill Sherman
Keyword(s):  
Task Performance ◽  
Model Validation ◽  
Predictive Models ◽  
Spatial Awareness ◽  
Level Flight ◽  
Helicopter Flight ◽  
Processing Resources ◽  
Multiple Resource ◽  
Multiple Task ◽  
Over Time

The objective of this research was to establish the validity of predictive models of workload in the context of a controlled simulation of a helicopter flight mission. The models that were evaluated contain increasing levels of sophistication regarding their assumptions about the competition for processing resources underlying multiple task performance. Ten subjects performed the simulation which involved various combinations of a low level flight task with three cognitive side tasks, pertaining to navigation, spatial awareness and computation. Side task information was delivered auditorily or visually. Results indicated that subjective workload is best predicted by relatively simple models that simply integrate the total demands of tasks over time (r = 0.65). In contrast, performance is not well predicted by these models (r < .10), but is best predicted by models that assume differential competition between processing resources (r = 0.47). The relevance of these data to predictive models and to the use of subjective measures for model validation is discussed.

La misura dell'attendibilitŕ dei modelli di previsione e l'unitŕ del sapere

FUTURIBILI ◽  
2009 ◽  
pp. 98-107
Author(s):  
Luciana Bozzo
Keyword(s):  
Urban Development ◽  
Predictive Model ◽  
Predictive Models ◽  
Territorial Planning ◽  
Private Citizens ◽  
Development Plans ◽  
Unity Of Knowledge ◽  
Science Cafe

- The reliability of predictive models is assured by the ability to establish a unity of knowledge, or rather of many branches of knowledge. This is the idea that leads the author to reflect on the prediction derived first of all from the "science café", defined as "a talking shop for scholars from a range of disciplines", who represent many branches of knowledge which are in fact a complete whole - "knowledge". The background for the predictive model discussed here is territorial planning, which encompasses an instrumental-explanatory component, a predictive component and an ideal. The construction of the predictive model and the degree of its reliability are produced by the process of unifying knowledge, and this confluence derives from knowledge of geographers, biologists, chemists, engineers, architects, agronomists, sociologists and private citizens. General Urban Development Plans stand as the instrumental and predictive model in which a certain unification of knowledge - at least operational - is achieved.

Predictive analytics of the technical condition of diesel locomotive systems using neural network predictive models

2021 ◽  
pp. 102-114
Author(s):  
Mikhail V. FEDOTOV ◽  
◽  
Vladimir V. GRACHEV ◽  
Keyword(s):  
Neural Network ◽  
Predictive Model ◽  
Predictive Models ◽  
Short Term Memory ◽  
Predictive Analytics ◽  
Practical Importance ◽  
Technical Condition ◽  
Feedforward Neural Network ◽  
Diesel Locomotive ◽  
Short Term

Objective: Study of the possibility of carrying out predictive analysis of the technical condition of locomotive equipment using neural network predictive models enabling to plan the scope of equipment maintenance for routine types of maintenance and repair. Methods: A comparative assessment of the accuracy of forecasts made using a feedforward neural network and a recurrent network with an LSTM layer (Long Short-Term Memory) has been carried out. For training and test-ing of predictive models, we used the results of monitoring the parameters of the lubrication sys-tem of the 2TE116 (2ТЭ116) diesel locomotive by means of on-board diagnostics. Results: The aver-age interval for preventive inspections (TO-3) of locomotives in the existing locomotive mainte-nance system is 25–30 days, and therefore it is this interval that determines the minimum duration of the lead-in period, which the predictive model should provide. We have established that a mod-el based on a feedforward neural network provides sufficient accuracy only for short-term fore-casts with a lead period of no more than 1–3 days. With a further increase in the lead-in period, the error of the model res¬ponse increases to 10–15 %, which prevents it from being effectively used for solving practical problems associated with planning the operation of service locomotive depots. At the same time, the ave¬rage response error of the predictive model based on a recurrent net-work with an LSTM layer does not exceed 3,5–5 % over a 30-day lead-in period, so it can be used to plan the scope and timing of locomotive maintenance procedures. Practical importance: The possi-bility of using time-series analysis methods for predictive analytics of the technical condition of units and systems of a locomotive is shown. Predictive models based on recurrent neural networks with LSTM layers provide prediction accuracy and lead-in period sufficient for solving practical prob-lems that are associated with planning the scope and timing of locomotive maintenance.

Development of a Simulation Model for Optimization of Business Process

2018 ◽  
pp. 26-42
Keyword(s):  
Simulation Model ◽  
Model Validation ◽  
Business Process ◽  
Research Question ◽  
Future Research ◽  
The Novel ◽  
Model Based ◽  
Model Validity ◽  
Proposed Model ◽  
New Research

Optimization of business process assists in efficient organization of business process. For the success of optimization of business process, a simulation model based on gap processes for the analysis of buyers' burstiness in business process has been proposed. However, the model has to be validated. The aim of the research is to implement a validation approach to the simulation model based on gap processes for the optimization of business process underpinning elaboration of a new research question on the model validity. The meaning of the key concepts of “validation,” “model validation,” and “model validation approach” is studied. The results of the present research show that the application of real system measurements validates the simulation model for the optimization of business process. The novel contribution of the manuscript is revealed in the newly created research question on the proposed model validity. Directions of future research are proposed.

Sign in / Sign up

Export Citation Format

Share Document