Toward a Better Understanding of Model Validation Metrics

2011 ◽  
Vol 133 (7) ◽  
Author(s):  
Yu Liu ◽  
Wei Chen ◽  
Paul Arendt ◽  
Hong-Zhong Huang
Keyword(s):  
Model Validation ◽  
Bayes Factor ◽  
Quantitative Measure ◽  
Model Parameters ◽  
Analysis And Design ◽  
Validation Metrics ◽  
Physical Experiments ◽  
Area Metric ◽  
Key Characteristics ◽  
Pros And Cons

Model validation metrics have been developed to provide a quantitative measure that characterizes the agreement between predictions and observations. In engineering design, the metrics become useful for model selection when alternative models are being considered. Additionally, the predictive capability of a computational model needs to be assessed before it is used in engineering analysis and design. Due to the various sources of uncertainties in both computer simulations and physical experiments, model validation must be conducted based on stochastic characteristics. Currently there is no unified validation metric that is widely accepted. In this paper, we present a classification of validation metrics based on their key characteristics along with a discussion of the desired features. Focusing on stochastic validation with the consideration of uncertainty in both predictions and physical experiments, four main types of metrics, namely classical hypothesis testing, Bayes factor, frequentist’s metric, and area metric, are examined to provide a better understanding of the pros and cons of each. Using mathematical examples, a set of numerical studies are designed to answer various research questions and study how sensitive these metrics are with respect to the experimental data size, the uncertainty from measurement error, and the uncertainty in unknown model parameters. The insight gained from this work provides useful guidelines for choosing the appropriate validation metric in engineering applications.

Some Metrics and a Bayesian Procedure for Validating Predictive Models in Engineering Design

2006 ◽  
Author(s):  
Wei Chen ◽  
Ying Xiong ◽  
Kwok-Leung Tsui ◽  
Shuchun Wang
Keyword(s):  
Engineering Design ◽  
Model Validation ◽  
Computer Model ◽  
Bayesian Approach ◽  
Model Prediction ◽  
Predictive Models ◽  
Design Decision ◽  
Validation Metrics ◽  
Model Based ◽  
Physical Experiments

Even though model-based simulations are widely used in engineering design, it remains a challenge to validate models and assess the risks and uncertainties associated with the use of predictive models for design decision making. In most of the existing work, model validation is viewed as verifying the model accuracy, measured by the agreement between computational and experimental results. However, from the design perspective, a good model is considered as the one that can provide the discrimination (good resolution) between design candidates. In this work, a Bayesian approach is presented to assess the uncertainty in model prediction by combining data from both physical experiments and the computer model. Based on the uncertainty quantification of model prediction, some design-oriented model validation metrics are further developed to guide designers for achieving high confidence of using predictive models in making a specific design decision. We demonstrate that the Bayesian approach provides a flexible framework for drawing inferences for predictions in the intended but may be untested design domain, where design settings of physical experiments and the computer model may or may not overlap. The implications of the proposed validation metrics are studied, and their potential roles in a model validation procedure are highlighted.

scholarly journals RNA Modification Level Estimation with pulseR

Genes ◽  
2018 ◽  
Vol 9 (12) ◽  
pp. 619
Author(s):  
Etienne Boileau ◽  
Christoph Dieterich
Keyword(s):  
Experimental Approach ◽  
High Efficiency ◽  
Rna Modification ◽  
Model Parameters ◽  
Metabolic Labeling ◽  
Rna Seq ◽  
Rna Modifications ◽  
Log Odds ◽  
Pros And Cons ◽  
Wide Scale

RNA modifications regulate the complex life of transcripts. An experimental approach called LAIC-seq was developed to characterize modification levels on a transcriptome-wide scale. In this method, the modified and unmodified molecules are separated using antibodies specific for a given RNA modification (e.g., m6A). In essence, the procedure of biochemical separation yields three fractions: Input, eluate, and supernatent, which are subjected to RNA-seq. In this work, we present a bioinformatics workflow, which starts from RNA-seq data to infer gene-specific modification levels by a statistical model on a transcriptome-wide scale. Our workflow centers around the pulseR package, which was originally developed for the analysis of metabolic labeling experiments. We demonstrate how to analyze data without external normalization (i.e., in the absence of spike-ins), given high efficiency of separation, and how, alternatively, scaling factors can be derived from unmodified spike-ins. Importantly, our workflow provides an estimate of uncertainty of modification levels in terms of confidence intervals for model parameters, such as gene expression and RNA modification levels. We also compare alternative model parametrizations, log-odds, or the proportion of the modified molecules and discuss the pros and cons of each representation. In summary, our workflow is a versatile approach to RNA modification level estimation, which is open to any read-count-based experimental approach.

Modelling Turbulent Flow in Deformable Highly Porous Seabed and Structures

2018 ◽  
Author(s):  
Hisham Elsafti ◽  
Hocine Oumeraci
Keyword(s):  
Porous Media ◽  
Transient Flow ◽  
Stokes Equations ◽  
Pore Fluid ◽  
Navier Stokes ◽  
Model Parameters ◽  
Navier Stokes Equations ◽  
Deformable Porous Media ◽  
Physical Experiments ◽  
Fluid Coupling

In this study, the fully-coupled and fully-dynamic Biot governing equations in the open-source geotechFoam solver are extended to account for pore fluid viscous stresses. Additionally, turbulent pore fluid flow in deformable porous media is modeled by means of the conventional eddy viscosity concept without the need to resolve all turbulence scales. A new approach is presented to account for porous media resistance to flow (solid-to-fluid coupling) by means of an effective viscosity, which accounts for tortuosity, grain shape and local turbulences induced by flow through porous media. The new model is compared to an implemented extended Darcy-Forchheimer model in the Navier-Stokes equations, which accounts for laminar, transitional, turbulent and transient flow regimes. Further, to account for skeleton deformation, the porosity and other model parameters are updated with regard to strain of geomaterials. The presented model is calibrated by means of available results of physical experiments of unidirectional and oscillatory flows.

Repeatability Assessment and Sensitivity Analysis of Needle Insertion Physical Experiment

2018 ◽  
Author(s):  
Murong Li ◽  
Yong Lei
Keyword(s):  
Sensitivity Analysis ◽  
Parameter Estimation ◽  
Model Validation ◽  
Ground Truth ◽  
Needle Insertion ◽  
Physical Experiment ◽  
Future Experiment ◽  
Parameter Uncertainties ◽  
Tissue Interactions ◽  
Physical Experiments

Needle insertion physical experiments are used as the ground truth for model validation and parameter estimation by measuring the needle defection and tissue deformation during the needle-tissue interactions. Hence parameter uncertainties can contribute experiment errors. To improve the repeatability and accuracy of such experiments, one-at-a-time (OAT) sensitivity analysis is used to study the impacts of the factors, such as stirring temperature, frozen time, thawing time during the process of making hydrogels as well as repeated path insertion and different puncture plane in the planer needle insertion experiments. The results show that the puncture plane has the greatest effect on the repeatability of needle insertion physic experiments, followed by repeated path insertion, while other factors have the least effect. The results serve to guide future experiment design for greater repeatability and accuracy.

scholarly journals Practice of Simulation and Life Cycle Assessment in Tribology—A Review

Materials ◽  
2020 ◽  
Vol 13 (16) ◽  
pp. 3489
Author(s):  
Abdulaziz Kurdi ◽  
Nahla Alhazmi ◽  
Hatem Alhazmi ◽  
Thamer Tabbakh
Keyword(s):  
Life Cycle Assessment ◽  
Life Cycle ◽  
Hierarchical Systems ◽  
Computing Power ◽  
Simulation Techniques ◽  
Physical Experiments ◽  
Complex Design ◽  
Pros And Cons ◽  
Recent Trends ◽  
Lubrication Conditions

To simulate today’s complex tribo-contact scenarios, a methodological breakdown of a complex design problem into simpler sub-problems is essential to achieve acceptable simulation outcomes. This also helps to manage iterative, hierarchical systems within given computational power. In this paper, the authors reviewed recent trends of simulation practices in tribology to model tribo-contact scenario and life cycle assessment (LCA) with the help of simulation. With the advancement of modern computers and computing power, increasing effort has been given towards simulation, which not only saves time and resources but also provides meaningful results. Having said that, like every other technique, simulation has some inherent limitations which need to be considered during practice. Keeping this in mind, the pros and cons of both physical experiments and simulation approaches are reviewed together with their interdependency and how one approach can benefit the other. Various simulation techniques are outlined with a focus on machine learning which will dominate simulation approaches in the future. In addition, simulation of tribo-contacts across different length scales and lubrication conditions is discussed in detail. An extension of the simulation approach, together with experimental data, can lead towards LCA of components which will provide us with a better understanding of the efficient usage of limited resources and conservation of both energy and resources.

scholarly journals Assessment of model validation outcomes of a new recursive spatial equilibrium model for the Greater Beijing

2017 ◽  
Vol 46 (5) ◽  
pp. 805-825 ◽  
Author(s):  
Li Wan ◽  
Ying Jin
Keyword(s):  
Model Validation ◽  
Equilibrium Model ◽  
Expert Assessment ◽  
Model Parameters ◽  
Multiple Time ◽  
Spatial Equilibrium ◽  
Beijing City ◽  
Spatial Equilibrium Model ◽  
Recursive Equilibrium ◽  
Validation Strategy

Robust calibration and validation of applied urban models are prerequisites for their successful, policy-cogent use. This is particularly important today when expert assessment is questioned and closely scrutinized. This paper proposes a new model calibration-validation strategy based on a spatial equilibrium model that incorporates multiple time horizons, such that the predictive capabilities of the model can be empirically tested. The model is implemented for the Greater Beijing city region and the model validation strategy is demonstrated over the Census years 2000 to 2010. Through forward/backward forecasting, the model validation helps to verify the stability of the model parameters as well as the predictive capabilities of the recursive equilibrium framework. The proposed modelling strategy sets a new standard for verifying and validating recursive equilibrium models. We also consider the wider implications of the approach.

A Hierarchical Model Validation of Predictive Models for Engineering Product Development

2009 ◽  
Author(s):  
Byeng D. Youn ◽  
Byung C. Jung ◽  
Zhimin Xi ◽  
Sang Bum Kim
Keyword(s):  
Model Validation ◽  
Computer Model ◽  
Hierarchical Model ◽  
Predictive Models ◽  
Likelihood Function ◽  
Hierarchical Level ◽  
Model Parameters ◽  
Bottom Up ◽  
Hierarchy Level ◽  
Statistical Calibration

As the role of predictive models has increased, the fidelity of computational results has been of great concern to engineering decision makers. Often our limited understanding of complex systems leads to building inappropriate predictive models. To address a growing concern about the fidelity of the predictive models, this paper proposes a hierarchical model validation procedure with two validation activities: (1) validation planning (top-down) and (2) validation execution (bottom-up). In the validation planning, engineers define either the physics-of-failure (PoF) mechanisms or the system performances of interest. Then, the engineering system is decomposed into subsystems or components of which computer models are partially valid in terms of PoF mechanisms or system performances of interest. Validation planning will identify vital tests and predictive models along with both known and unknown model parameter(s). The validation execution takes a bottom-up approach, improving the fidelity of the computer model at any hierarchical level using a statistical calibration technique. This technique compares the observed test results with the predicted results from the computer model. A likelihood function is used for the comparison metric. In the statistical calibration, an optimization technique is employed to maximize the likelihood function while determining the unknown model parameters. As the predictive model at a lower hierarchy level becomes valid, the valid model is fused into a model at a higher hierarchy level. The validation execution is then continued for the model at the higher hierarchy level. A cellular phone is used to demonstrate the hierarchical validation of predictive models presented in this paper.

scholarly journals Validation Metrics for Deterministic and Probabilistic Data

2018 ◽  
Vol 3 (3) ◽  
Author(s):  
Kathryn A. Maupin ◽  
Laura P. Swiler ◽  
Nathan W. Porter
Keyword(s):  
Computational Models ◽  
Modern Science ◽  
Probabilistic Data ◽  
Validation Metrics ◽  
Physical Experiments ◽  
History Of ◽  
Computational Modeling And Simulation ◽  
History Of Use ◽  
The Impact ◽  
Application Specific

Computational modeling and simulation are paramount to modern science. Computational models often replace physical experiments that are prohibitively expensive, dangerous, or occur at extreme scales. Thus, it is critical that these models accurately represent and can be used as replacements for reality. This paper provides an analysis of metrics that may be used to determine the validity of a computational model. While some metrics have a direct physical meaning and a long history of use, others, especially those that compare probabilistic data, are more difficult to interpret. Furthermore, the process of model validation is often application-specific, making the procedure itself challenging and the results difficult to defend. We therefore provide guidance and recommendations as to which validation metric to use, as well as how to use and decipher the results. An example is included that compares interpretations of various metrics and demonstrates the impact of model and experimental uncertainty on validation processes.

A Component-Based Parametric Reduced-Order Modeling Technique and Its Application to Probabilistic Vibration Analysis and Design Optimization

2006 ◽  
Author(s):  
Keychun Park ◽  
Geng Zhang ◽  
Matthew P. Castanier ◽  
Christophe Pierre
Keyword(s):  
Design Optimization ◽  
Probabilistic Analysis ◽  
Structural Design ◽  
Vibration Analysis ◽  
Reduced Order Modeling ◽  
Design Parameters ◽  
Model Parameters ◽  
Reduced Order ◽  
Analysis And Design ◽  
Structural Design Optimization

In this paper, a component-based parametric reduced-order modeling (PROM) technique for vibration analysis of complex structures is presented, and applications to both structural design optimization and uncertainty analysis are shown. In structural design optimization, design parameters are allowed to vary in the feasible design space. In probabilistic analysis, selected model parameters are assumed to have predefined probability distributions. For both cases, each realization corresponding to a specific set of parameter values could be evaluated accurately based on the exact modes for the system with those parametric values. However, as the number of realizations increases, this approach becomes prohibitively expensive, especially for largescale finite element models. Recently, a PROM method that employs a fixed projection basis was introduced to avoid the eigenanalysis for each variation while retaining good accuracy. The fixed basis is comprised of a combination of selected mode sets of the full model calculated at only a few sampling points in the parameter space. However, the preparation for the basis may still be cumbersome, and the simulation cost and the model size increase rapidly as the number of parameters increases. In this work, a component-based approach is taken to improve the efficiency and effectiveness of the PROM technique. In particular, a component mode synthesis method is employed so that the parameter changes are captured at the substructure level and the analysis procedure is accelerated. Numerical results are presented for two example problems, a design optimization of a pickup truck and a probabilistic analysis of a simple L-shaped plate. It is shown that the new component-based approach significantly improves the efficiency of the PROM technique.

Tracer Technique to Measure in Vivo Chemical Transport Rates within an Implantable Cell Transplantation Device

1995 ◽  
Vol 4 (2) ◽  
pp. 201-217 ◽  
Author(s):  
Jeffrey G. Sarver ◽  
Ronald L. Fournier ◽  
Peter J. Goldblatt ◽  
Tamara L. Phares ◽  
Sara E. Mertz ◽  
...  
Keyword(s):  
Cell Transplantation ◽  
Circulatory System ◽  
Quantitative Measure ◽  
Chemical Transport ◽  
Porous Matrix ◽  
The Body ◽  
Model Parameters ◽  
Tracer Technique ◽  
Capillary Growth

An in vivo tracer technique that uses radiolabeled inulin as the tracer molecule has been developed to assess the rate of chemical transport between the cell transplantation chamber of an implantable bioartificial device and the host's circulatory system. The device considered here employs site-directed neovascularization of a porous matrix to induce capillary growth adjacent to an immunoisolated cell implantation chamber. This device design is being investigated as a vehicle for therapeutic cell transplantation, with the advantages that it allows the cells to perform their therapeutic function without the danger of immune rejection and it avoids damaging contact of blood flow with artificial surfaces. A pharmacokinetic model of the mass transport between the implantation chamber, the vascularized matrix, and the body has been devised to allow proper analysis and understanding of the experimental tracer results. Experiments performed in this study have been principally directed at evaluation of the tracer model parameters, but results also provide a quantitative measure of the progression of capillary growth into a porous matrix. Measured plasma tracer levels demonstrate that chemical transport rates within the implanted device increase with the progression of matrix vascular ingrowth. Agreement between the fitted model curves and the corresponding measured concentrations at different levels of capillary ingrowth demonstrate that the model provides a realistic representation of the actual capillary-mediated transport phenomena occurring within the device.

Sign in / Sign up

Export Citation Format

Share Document