A Hierarchical Model Validation of Predictive Models for Engineering Product Development

2009 ◽  
Author(s):  
Byeng D. Youn ◽  
Byung C. Jung ◽  
Zhimin Xi ◽  
Sang Bum Kim
Keyword(s):  
Model Validation ◽  
Computer Model ◽  
Hierarchical Model ◽  
Predictive Models ◽  
Likelihood Function ◽  
Hierarchical Level ◽  
Model Parameters ◽  
Bottom Up ◽  
Hierarchy Level ◽  
Statistical Calibration

As the role of predictive models has increased, the fidelity of computational results has been of great concern to engineering decision makers. Often our limited understanding of complex systems leads to building inappropriate predictive models. To address a growing concern about the fidelity of the predictive models, this paper proposes a hierarchical model validation procedure with two validation activities: (1) validation planning (top-down) and (2) validation execution (bottom-up). In the validation planning, engineers define either the physics-of-failure (PoF) mechanisms or the system performances of interest. Then, the engineering system is decomposed into subsystems or components of which computer models are partially valid in terms of PoF mechanisms or system performances of interest. Validation planning will identify vital tests and predictive models along with both known and unknown model parameter(s). The validation execution takes a bottom-up approach, improving the fidelity of the computer model at any hierarchical level using a statistical calibration technique. This technique compares the observed test results with the predicted results from the computer model. A likelihood function is used for the comparison metric. In the statistical calibration, an optimization technique is employed to maximize the likelihood function while determining the unknown model parameters. As the predictive model at a lower hierarchy level becomes valid, the valid model is fused into a model at a higher hierarchy level. The validation execution is then continued for the model at the higher hierarchy level. A cellular phone is used to demonstrate the hierarchical validation of predictive models presented in this paper.

Some Metrics and a Bayesian Procedure for Validating Predictive Models in Engineering Design

2006 ◽  
Author(s):  
Wei Chen ◽  
Ying Xiong ◽  
Kwok-Leung Tsui ◽  
Shuchun Wang
Keyword(s):  
Engineering Design ◽  
Model Validation ◽  
Computer Model ◽  
Bayesian Approach ◽  
Model Prediction ◽  
Predictive Models ◽  
Design Decision ◽  
Validation Metrics ◽  
Model Based ◽  
Physical Experiments

Even though model-based simulations are widely used in engineering design, it remains a challenge to validate models and assess the risks and uncertainties associated with the use of predictive models for design decision making. In most of the existing work, model validation is viewed as verifying the model accuracy, measured by the agreement between computational and experimental results. However, from the design perspective, a good model is considered as the one that can provide the discrimination (good resolution) between design candidates. In this work, a Bayesian approach is presented to assess the uncertainty in model prediction by combining data from both physical experiments and the computer model. Based on the uncertainty quantification of model prediction, some design-oriented model validation metrics are further developed to guide designers for achieving high confidence of using predictive models in making a specific design decision. We demonstrate that the Bayesian approach provides a flexible framework for drawing inferences for predictions in the intended but may be untested design domain, where design settings of physical experiments and the computer model may or may not overlap. The implications of the proposed validation metrics are studied, and their potential roles in a model validation procedure are highlighted.

TASKILLAN: A Simulation to Predict the Validity of Multiple Resource Models of Aviation Workload

1988 ◽  
Vol 32 (2) ◽  
pp. 168-172 ◽  
Author(s):  
Christopher D. Wickens ◽  
Kelly Harwood ◽  
Leon Segal ◽  
Inge Tkalcevic ◽  
Bill Sherman
Keyword(s):  
Task Performance ◽  
Model Validation ◽  
Predictive Models ◽  
Spatial Awareness ◽  
Level Flight ◽  
Helicopter Flight ◽  
Processing Resources ◽  
Multiple Resource ◽  
Multiple Task ◽  
Over Time

The objective of this research was to establish the validity of predictive models of workload in the context of a controlled simulation of a helicopter flight mission. The models that were evaluated contain increasing levels of sophistication regarding their assumptions about the competition for processing resources underlying multiple task performance. Ten subjects performed the simulation which involved various combinations of a low level flight task with three cognitive side tasks, pertaining to navigation, spatial awareness and computation. Side task information was delivered auditorily or visually. Results indicated that subjective workload is best predicted by relatively simple models that simply integrate the total demands of tasks over time (r = 0.65). In contrast, performance is not well predicted by these models (r < .10), but is best predicted by models that assume differential competition between processing resources (r = 0.47). The relevance of these data to predictive models and to the use of subjective measures for model validation is discussed.

scholarly journals Bayesian calibration of terrestrial ecosystem models: a study of advanced Markov chain Monte Carlo methods

2017 ◽  
Vol 14 (18) ◽  
pp. 4295-4314 ◽  
Author(s):  
Dan Lu ◽  
Daniel Ricciuto ◽  
Anthony Walker ◽  
Cosmin Safta ◽  
William Munger
Keyword(s):  
Monte Carlo ◽  
Markov Chain ◽  
Markov Chain Monte Carlo ◽  
Likelihood Function ◽  
Terrestrial Ecosystem ◽  
Error Model ◽  
Model Parameters ◽  
Posterior Distributions ◽  
Ecosystem Models ◽  
Bayesian Calibration

Abstract. Calibration of terrestrial ecosystem models is important but challenging. Bayesian inference implemented by Markov chain Monte Carlo (MCMC) sampling provides a comprehensive framework to estimate model parameters and associated uncertainties using their posterior distributions. The effectiveness and efficiency of the method strongly depend on the MCMC algorithm used. In this work, a differential evolution adaptive Metropolis (DREAM) algorithm is used to estimate posterior distributions of 21 parameters for the data assimilation linked ecosystem carbon (DALEC) model using 14 years of daily net ecosystem exchange data collected at the Harvard Forest Environmental Measurement Site eddy-flux tower. The calibration of DREAM results in a better model fit and predictive performance compared to the popular adaptive Metropolis (AM) scheme. Moreover, DREAM indicates that two parameters controlling autumn phenology have multiple modes in their posterior distributions while AM only identifies one mode. The application suggests that DREAM is very suitable to calibrate complex terrestrial ecosystem models, where the uncertain parameter size is usually large and existence of local optima is always a concern. In addition, this effort justifies the assumptions of the error model used in Bayesian calibration according to the residual analysis. The result indicates that a heteroscedastic, correlated, Gaussian error model is appropriate for the problem, and the consequent constructed likelihood function can alleviate the underestimation of parameter uncertainty that is usually caused by using uncorrelated error models.

scholarly journals Efficient detection of repeating sites to accelerate phylogenetic likelihood calculations

2016 ◽  
Author(s):  
Kassian Kobert ◽  
Alexandros Stamatakis ◽  
Tomáš Flouri
Keyword(s):  
Evolutionary Biology ◽  
Likelihood Function ◽  
Simulated Data ◽  
Evolutionary Model ◽  
Identical Result ◽  
Model Parameters ◽  
Data Sets ◽  
Efficient Detection ◽  
Novel Method ◽  
Computational Bottleneck

The phylogenetic likelihood function is the major computational bottleneck in several applications of evolutionary biology such as phylogenetic inference, species delimitation, model selection and divergence times estimation. Given the alignment, a tree and the evolutionary model parameters, the likelihood function computes the conditional likelihood vectors for every node of the tree. Vector entries for which all input data are identical result in redundant likelihood operations which, in turn, yield identical conditional values. Such operations can be omitted for improving run-time and, using appropriate data structures, reducing memory usage. We present a fast, novel method for identifying and omitting such redundant operations in phylogenetic likelihood calculations, and assess the performance improvement and memory saving attained by our method. Using empirical and simulated data sets, we show that a prototype implementation of our method yields up to 10-fold speedups and uses up to 78% less memory than one of the fastest and most highly tuned implementations of the phylogenetic likelihood function currently available. Our method is generic and can seamlessly be integrated into any phylogenetic likelihood implementation.

Model Expansion in Model-Driven Architectures

2012 ◽  
pp. 28-50
Author(s):  
Tsung Lee ◽  
Jhih-Syan Hou
Keyword(s):  
Hierarchical Model ◽  
Expansion Method ◽  
Model Parameters ◽  
Model Composition ◽  
Tree Structures ◽  
Model Driven ◽  
Design Structure ◽  
Model Instance ◽  
Composition Graph ◽  
Model Expansion

In this chapter, the authors introduce a model expansion method that is used in a new methodology of model composition and evolution for broad design domains. In the methodology, hierarchical model compositional relationships are captured in a model composition graph (MCG) as a schema of designs. An MCG schema can be used as a blueprint for systematic and flexible evolution of designs with three hierarchical model refinement operations: expansion, synthesis, and configuration. In this methodology, due to the need of hierarchical sharing in software and hardware domains, the authors designed an algorithm to achieve conditional and recursive model expansion with hierarchical model instance sharing that is not achievable in other expansion methods. Hierarchical model instance sharing complicates the design structure from tree structures to graph structures. The model expansion algorithm was thus designed with enhanced features of maintenance of MCG instance consistency, path-based search of shared submodel instances, and dependency preserving expansion ordering. The expansion specification and the expansion process are integrated with the MCG-based methodology. Model parameters set by designers and other refinement operations can be used to guide each expansion step of design models iteratively.

scholarly journals Generalized Exponentiated Weibull Linear Model in the Presence of Covariates

2017 ◽  
Vol 6 (3) ◽  
pp. 75
Author(s):  
Tiago V. F. Santana ◽  
Edwin M. M. Ortega ◽  
Gauss M. Cordeiro ◽  
Adriano K. Suzuki
Keyword(s):  
Regression Model ◽  
Linear Model ◽  
Goodness Of Fit ◽  
Likelihood Function ◽  
Confidence Bands ◽  
Model Parameters ◽  
Random Component ◽  
Monte Carlo Simulation Study ◽  
New Model ◽  
Exponentiated Weibull

A new regression model based on the exponentiated Weibull with the structure distribution and the structure of the generalized linear model, called the generalized exponentiated Weibull linear model (GEWLM), is proposed. The GEWLM is composed by three important structural parts: the random component, characterized by the distribution of the response variable; the systematic component, which includes the explanatory variables in the model by means of a linear structure; and a link function, which connects the systematic and random parts of the model. Explicit expressions for the logarithm of the likelihood function, score vector and observed and expected information matrices are presented. The method of maximum likelihood and a Bayesian procedure are adopted for estimating the model parameters. To detect influential observations in the new model, we use diagnostic measures based on the local influence and Bayesian case influence diagnostics. Also, we show that the estimates of the GEWLM are  robust to deal with the presence of outliers in the data. Additionally, to check whether the model supports its assumptions, to detect atypical observations and to verify the goodness-of-fit of the regression model, we define residuals based on the quantile function, and perform a Monte Carlo simulation study to construct confidence bands from the generated envelopes. We apply the new model to a dataset from the insurance area.

scholarly journals Assessment of model validation outcomes of a new recursive spatial equilibrium model for the Greater Beijing

2017 ◽  
Vol 46 (5) ◽  
pp. 805-825 ◽  
Author(s):  
Li Wan ◽  
Ying Jin
Keyword(s):  
Model Validation ◽  
Equilibrium Model ◽  
Expert Assessment ◽  
Model Parameters ◽  
Multiple Time ◽  
Spatial Equilibrium ◽  
Beijing City ◽  
Spatial Equilibrium Model ◽  
Recursive Equilibrium ◽  
Validation Strategy

Robust calibration and validation of applied urban models are prerequisites for their successful, policy-cogent use. This is particularly important today when expert assessment is questioned and closely scrutinized. This paper proposes a new model calibration-validation strategy based on a spatial equilibrium model that incorporates multiple time horizons, such that the predictive capabilities of the model can be empirically tested. The model is implemented for the Greater Beijing city region and the model validation strategy is demonstrated over the Census years 2000 to 2010. Through forward/backward forecasting, the model validation helps to verify the stability of the model parameters as well as the predictive capabilities of the recursive equilibrium framework. The proposed modelling strategy sets a new standard for verifying and validating recursive equilibrium models. We also consider the wider implications of the approach.

A Comparison of Numerical Optimizers in Developing High Dimensional Surrogate Models

2019 ◽  
Author(s):  
Yanwen Xu ◽  
Pingfeng Wang
Keyword(s):  
Gaussian Process ◽  
Process Model ◽  
Likelihood Function ◽  
Surrogate Models ◽  
Kernel Functions ◽  
Function Optimization ◽  
High Dimensional ◽  
Model Parameters ◽  
Gradient Based ◽  
Gp Model

Abstract The Gaussian Process (GP) model has become one of the most popular methods to develop computationally efficient surrogate models in many engineering design applications, including simulation-based design optimization and uncertainty analysis. When more observations are used for high dimensional problems, estimating the best model parameters of Gaussian Process model is still an essential yet challenging task due to considerable computation cost. One of the most commonly used methods to estimate model parameters is Maximum Likelihood Estimation (MLE). A common bottleneck arising in MLE is computing a log determinant and inverse over a large positive definite matrix. In this paper, a comparison of five commonly used gradient based and non-gradient based optimizers including Sequential Quadratic Programming (SQP), Quasi-Newton method, Interior Point method, Trust Region method and Pattern Line Search for likelihood function optimization of high dimension GP surrogate modeling problem is conducted. The comparison has been focused on the accuracy of estimation, the efficiency of computation and robustness of the method for different types of Kernel functions.

scholarly journals Probabilistic Updating of Structural Models for Damage Assessment Using Approximate Bayesian Computation

Sensors ◽  
2020 ◽  
Vol 20 (11) ◽  
pp. 3197 ◽  
Author(s):  
Zhouquan Feng ◽  
Yang Lin ◽  
Wenzan Wang ◽  
Xugang Hua ◽  
Zhengqing Chen
Keyword(s):  
Damage Assessment ◽  
Approximate Bayesian Computation ◽  
Model Updating ◽  
Likelihood Function ◽  
Probabilistic Approach ◽  
Model Parameters ◽  
Bayesian Computation ◽  
The Novel ◽  
Subset Simulation ◽  
Approximate Bayesian

A novel probabilistic approach for model updating based on approximate Bayesian computation with subset simulation (ABC-SubSim) is proposed for damage assessment of structures using modal data. The ABC-SubSim is a likelihood-free Bayesian approach in which the explicit expression of likelihood function is avoided and the posterior samples of model parameters are obtained using the technique of subset simulation. The novel contributions of this paper are on three fronts: one is the introduction of some new stopping criteria to find an appropriate tolerance level for the metric used in the ABC-SubSim; the second one is the employment of a hybrid optimization scheme to find finer optimal values for the model parameters; and the last one is the adoption of an iterative approach to determine the optimal weighting factors related to the residuals of modal frequency and mode shape in the metric. The effectiveness of this approach is demonstrated using three illustrative examples.

Updating Predictive Models: Calibration, Bias Correction and Identifiability

2010 ◽  
Author(s):  
Paul D. Arendt ◽  
Wei Chen ◽  
Daniel W. Apley
Keyword(s):  
Computer Model ◽  
Bayesian Approach ◽  
Bias Correction ◽  
Process Model ◽  
Model Updating ◽  
Future Research ◽  
Model Parameters ◽  
Sources Of Uncertainty ◽  
Uncertain Model ◽  
Many Sources

Model updating, which utilizes mathematical means to combine model simulations with physical observations for improving model predictions, has been viewed as an integral part of a model validation process. While calibration is often used to “tune” uncertain model parameters, bias-correction has been used to capture model inadequacy due to a lack of knowledge of the physics of a problem. While both sources of uncertainty co-exist, these two techniques are often implemented separately in model updating. This paper examines existing approaches to model updating and presents a modular Bayesian approach as a comprehensive framework that accounts for many sources of uncertainty in a typical model updating process and provides stochastic predictions for the purpose of design. In addition to the uncertainty in the computer model parameters and the computer model itself, this framework accounts for the experimental uncertainty and the uncertainty due to the lack of data in both computer simulations and physical experiments using the Gaussian process model. Several challenges are apparent in the implementation of the modular Bayesian approach. We argue that distinguishing between uncertain model parameters (calibration) and systematic inadequacies (bias correction) is often quite challenging due to an identifiability issue. We present several explanations and examples of this issue and bring up the needs of future research in distinguishing between the two sources of uncertainty.

Sign in / Sign up

Export Citation Format

Share Document