An Adaptive Bayesian Sequential Sampling Approach for Global Metamodeling

2015 ◽  
Vol 138 (1) ◽  
Author(s):  
Haitao Liu ◽  
Shengli Xu ◽  
Ying Ma ◽  
Xudong Chen ◽  
Xiaofang Wang
Keyword(s):  
Sequential Sampling ◽  
Computer Experiments ◽  
Simulation Models ◽  
Search Pattern ◽  
Clustering Problem ◽  
Bayesian Sampling ◽  
Cross Validation Error ◽  
Local Space ◽  
Leave One Out ◽  
Global Metamodeling

Computer simulations have been increasingly used to study physical problems in various fields. To relieve computational budgets, the cheap-to-run metamodels, constructed from finite experiment points in the design space using the design of computer experiments (DOE), are employed to replace the costly simulation models. A key issue related to DOE is designing sequential computer experiments to achieve an accurate metamodel with as few points as possible. This article investigates the performance of current Bayesian sampling approaches and proposes an adaptive maximum entropy (AME) approach. In the proposed approach, the leave-one-out (LOO) cross-validation error estimates the error information in an easy way, the local space-filling exploration strategy avoids the clustering problem, and the search pattern from global to local improves the sampling efficiency. A comparison study of six examples with different types of initial points demonstrated that the AME approach is very promising for global metamodeling.

scholarly journals Statistical Tests for Cross-Validation of Kriging Models

2021 ◽  
Author(s):  
Jack P. C. Kleijnen ◽  
Wim C. M. van Beers
Keyword(s):  
Monte Carlo ◽  
Cross Validation ◽  
Statistical Tests ◽  
Computer Experiments ◽  
Simulation Models ◽  
Statistical Test ◽  
Type I ◽  
Monte Carlo Experiments ◽  
Leave One Out ◽  
Kriging Predictor

Kriging or Gaussian process models are popular metamodels (surrogate models or emulators) of simulation models; these metamodels give predictors for input combinations that are not simulated. To validate these metamodels for computationally expensive simulation models, the analysts often apply computationally efficient cross-validation. In this paper, we derive new statistical tests for so-called leave-one-out cross-validation. Graphically, we present these tests as scatterplots augmented with confidence intervals that use the estimated variances of the Kriging predictors. To estimate the true variances of these predictors, we might use bootstrapping. Like other statistical tests, our tests—with or without bootstrapping—have type I and type II error probabilities; to estimate these probabilities, we use Monte Carlo experiments. We also use such experiments to investigate statistical convergence. To illustrate the application of our tests, we use (i) an example with two inputs and (ii) the popular borehole example with eight inputs. Summary of Contribution: Simulation models are very popular in operations research (OR) and are also known as computer simulations or computer experiments. A popular topic is design and analysis of computer experiments. This paper focuses on Kriging methods and cross-validation methods applied to simulation models; these methods and models are often applied in OR. More specifically, the paper provides the following; (1) the basic variant of a new statistical test for leave-one–out cross-validation; (2) a bootstrap method for the estimation of the true variance of the Kriging predictor; and (3) Monte Carlo experiments for the evaluation of the consistency of the Kriging predictor, the convergence of the Studentized prediction error to the standard normal variable, and the convergence of the expected experimentwise type I error rate to the prespecified nominal value. The new statistical test is illustrated through examples, including the popular borehole model.

A Robust Error-Pursuing Sequential Sampling Approach for Global Metamodeling Based on Voronoi Diagram and Cross Validation

2014 ◽  
Vol 136 (7) ◽  
Author(s):  
Shengli Xu ◽  
Haitao Liu ◽  
Xiaofang Wang ◽  
Xiaomo Jiang
Keyword(s):  
Engineering Design ◽  
Voronoi Diagram ◽  
Cross Validation ◽  
Sequential Sampling ◽  
Voronoi Cell ◽  
Sensitive Region ◽  
Engineering Design Problems ◽  
Simulation Based ◽  
Global Metamodeling ◽  
Sampling Approach

Surrogate models are widely used in simulation-based engineering design and optimization to save the computing cost. The choice of sampling approach has a great impact on the metamodel accuracy. This article presents a robust error-pursuing sequential sampling approach called cross-validation (CV)-Voronoi for global metamodeling. During the sampling process, CV-Voronoi uses Voronoi diagram to partition the design space into a set of Voronoi cells according to existing points. The error behavior of each cell is estimated by leave-one-out (LOO) cross-validation approach. Large prediction error indicates that the constructed metamodel in this Voronoi cell has not been fitted well and, thus, new points should be sampled in this cell. In order to rapidly improve the metamodel accuracy, the proposed approach samples a Voronoi cell with the largest error value, which is marked as a sensitive region. The sampling approach exploits locally by the identification of sensitive region and explores globally with the shift of sensitive region. Comparative results with several sequential sampling approaches have demonstrated that the proposed approach is simple, robust, and achieves the desired metamodel accuracy with fewer samples, that is needed in simulation-based engineering design problems.

scholarly journals ASAMS: An Adaptive Sequential Sampling and Automatic Model Selection for Artificial Intelligence Surrogate Modeling

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5332
Author(s):  
Carlos A. Duchanoy ◽  
Hiram Calvo ◽  
Marco A. Moreno-Armendáriz
Keyword(s):  
Artificial Intelligence ◽  
Model Selection ◽  
Adaptive Sampling ◽  
Search Algorithm ◽  
Sequential Sampling ◽  
Surrogate Modeling ◽  
Kriging Model ◽  
Surrogate Models ◽  
Physical Experiments ◽  
Leave One Out

Surrogate Modeling (SM) is often used to reduce the computational burden of time-consuming system simulations. However, continuous advances in Artificial Intelligence (AI) and the spread of embedded sensors have led to the creation of Digital Twins (DT), Design Mining (DM), and Soft Sensors (SS). These methodologies represent a new challenge for the generation of surrogate models since they require the implementation of elaborated artificial intelligence algorithms and minimize the number of physical experiments measured. To reduce the assessment of a physical system, several existing adaptive sequential sampling methodologies have been developed; however, they are limited in most part to the Kriging models and Kriging-model-based Monte Carlo Simulation. In this paper, we integrate a distinct adaptive sampling methodology to an automated machine learning methodology (AutoML) to help in the process of model selection while minimizing the system evaluation and maximizing the system performance for surrogate models based on artificial intelligence algorithms. In each iteration, this framework uses a grid search algorithm to determine the best candidate models and perform a leave-one-out cross-validation to calculate the performance of each sampled point. A Voronoi diagram is applied to partition the sampling region into some local cells, and the Voronoi vertexes are considered as new candidate points. The performance of the sample points is used to estimate the accuracy of the model for a set of candidate points to select those that will improve more the model’s accuracy. Then, the number of candidate models is reduced. Finally, the performance of the framework is tested using two examples to demonstrate the applicability of the proposed method.

On Sequential Sampling for Global Metamodeling in Engineering Design

2002 ◽  
Author(s):  
Ruichen Jin ◽  
Wei Chen ◽  
Agus Sudjianto
Keyword(s):  
Complex Systems ◽  
Engineering Design ◽  
Sequential Sampling ◽  
Test Problems ◽  
Sampling Strategies ◽  
General Applicability ◽  
One Stage ◽  
Computationally Expensive ◽  
The One ◽  
Global Metamodeling

Approximation models (also known as metamodels) have been widely used in engineering design to facilitate analysis and optimization of complex systems that involve computationally expensive simulation programs. The accuracy of metamodels is directly related to the sampling strategies used. Our goal in this paper is to investigate the general applicability of sequential sampling for creating global metamodels. Various sequential sampling approaches are reviewed and new approaches are proposed. The performances of these approaches are investigated against that of the one-stage approach using a set of test problems with a variety of features. The potential usages of sequential sampling strategies are also discussed.

Constructing Surrogate Models for Springback in U-Bending Process

2012 ◽  
Vol 445 ◽  
pp. 177-182
Author(s):  
Deniz Bekar ◽  
Erdem Acar ◽  
Firat Ozer ◽  
Mehmet Ali Guler
Keyword(s):  
Sheet Thickness ◽  
Side Wall ◽  
Latin Hypercube Sampling ◽  
Simulation Models ◽  
Surrogate Models ◽  
Training Data ◽  
Thickness Reduction ◽  
Bending Process ◽  
Polynomial Response Surface ◽  
Leave One Out

In this study, surrogate models are constructed to approximate the behavior of simulation models for springback angles, sidewall curl, and sheet thickness reduction in U-bending process. The surrogate-modeling techniques used here are: (i) polynomial response surface (PRS), (ii) Kriging (KR) and (iii) radial basis functions (RBF). While constructing surrogate models, the following procedure is pursued. First, a set of training points is generated using Latin hypercube sampling method, and finite element simulations are performed at these points. Then, surrogate models are constructed utilizing the training data. The accuracies of the surrogate models are evaluated by using the leave-one-out cross validation errors. First-order PRS is found to be most accurate surrogate model for prediction of the springback angles, side wall curl, and sheet thickness reduction.

scholarly journals BEPU robustness analysis via perturbed law-based sensitivity indices

2021 ◽  
pp. 1748006X2110365
Author(s):  
Bertrand Iooss ◽  
Vanessa Vergès ◽  
Vincent Larget
Keyword(s):  
Nuclear Reactor ◽  
Probability Distributions ◽  
Safety Margin ◽  
Robustness Analysis ◽  
Computer Experiments ◽  
Simulation Models ◽  
Pressurized Water ◽  
Sensitivity Indices ◽  
Input Variables ◽  
The Impact

The “best-estimate plus uncertainty” (BEPU) methodology is the term used in the nuclear engineering community when dealing with uncertainty quantification issues in realistic numerical simulation models. One of the most critical hypothesis in these studies is the choice of the probability distributions of uncertain input variables which are propagated through the model. Bringing stringent justifications to the BEPU approach, especially in a safety study, requires quantifying the impact of potential uncertainty on the input variable distribution. To solve this problem, this paper deepens the robustness analysis based on the “Perturbed Law-based sensitivity Indices” (PLI). The PLI quantifies the impact of a perturbation of an input distribution on the quantity of interest (as a quantile of a model output or a safety margin) in the BEPU study. The mathematical formalism of the PLI is applied to two particular quantities of interest: the quantile and the superquantile. For both quantities, the PLI can be easily computed using a unique Monte-Carlo sample containing model inputs and output. Numerical tests are developed in order to define validity criteria of a PLI-based robustness analysis. The practical use of the method is illustrated on thermal-hydraulic computer experiments, simulating a cold leg Intermediate Break Loss Of Coolant Accident (IBLOCA) in a pressurized water nuclear reactor.

scholarly journals A rainfall forecasting method using machine learning models and its application to the Fukuoka city case

2012 ◽  
Vol 22 (4) ◽  
pp. 841-854 ◽  
Author(s):  
S. Monira Sumi ◽  
M. Faisal Zaman ◽  
Hideo Hirose
Keyword(s):  
Machine Learning ◽  
Moving Average ◽  
Principal Component ◽  
Processing Technique ◽  
Monthly Rainfall ◽  
Multivariate Adaptive Regression Splines ◽  
Support Vector ◽  
Rainfall Forecasting ◽  
Cross Validation Error ◽  
Leave One Out

In the present article, an attempt is made to derive optimal data-driven machine learning methods for forecasting an average daily and monthly rainfall of the Fukuoka city in Japan. This comparative study is conducted concentrating on three aspects: modelling inputs, modelling methods and pre-processing techniques. A comparison between linear correlation analysis and average mutual information is made to find an optimal input technique. For the modelling of the rainfall, a novel hybrid multi-model method is proposed and compared with its constituent models. The models include the artificial neural network, multivariate adaptive regression splines, the k-nearest neighbour, and radial basis support vector regression. Each of these methods is applied to model the daily and monthly rainfall, coupled with a pre-processing technique including moving average and principal component analysis. In the first stage of the hybrid method, sub-models from each of the above methods are constructed with different parameter settings. In the second stage, the sub-models are ranked with a variable selection technique and the higher ranked models are selected based on the leave-one-out cross-validation error. The forecasting of the hybrid model is performed by the weighted combination of the finally selected models.

Making the Most Out of Surrogate Models: Tricks of the Trade

2010 ◽  
Author(s):  
Felipe A. C. Viana ◽  
Christian Gogu ◽  
Raphael T. Haftka
Keyword(s):  
Design Automation ◽  
Sequential Sampling ◽  
Surrogate Modeling ◽  
Computer Experiments ◽  
Limited Resources ◽  
High Fidelity ◽  
Statistical Tools ◽  
Computational Burden ◽  
Intensive Research ◽  
Variable Reduction

Design analysis and optimization based on high-fidelity computer experiments is commonly expensive. Surrogate modeling is often the tool of choice for reducing the computational burden. However, even after years of intensive research, surrogate modeling still involves a struggle to achieve maximum accuracy within limited resources. This work summarizes advanced and yet simple statistical tools that help. We focus on four techniques with increasing popularity in the design automation community: (i) screening and variable reduction in both the input and the output spaces, (ii) simultaneous use of multiple surrogates, (iii) sequential sampling and optimization, and (iv) conservative estimators.

Sign in / Sign up

Export Citation Format

Share Document