Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem

2016 ◽  
Vol 1 (1) ◽  
Author(s):  
Wei Li ◽  
Shishi Chen ◽  
Zhen Jiang ◽  
Daniel W. Apley ◽  
Zhenzhou Lu ◽  
...  
Keyword(s):  
Machine Learning ◽  
Predictive Model ◽  
Bias Correction ◽  
Verification And Validation ◽  
Model Parameters ◽  
Loading Conditions ◽  
Multiple Sources ◽  
Bayesian Calibration ◽  
Sources Of Uncertainty ◽  
Machine Learning Model

This paper describes an integrated Bayesian calibration, bias correction, and machine learning approach to the validation challenge problem posed at the Sandia Verification and Validation Challenge Workshop, May 7–9, 2014. Three main challenges are recognized as: I—identification of unknown model parameters; II—quantification of multiple sources of uncertainty; and III—validation assessment when there are no direct experimental measurements associated with one of the quantities of interest (QoIs), i.e., the von Mises stress. This paper addresses these challenges as follows. For challenge I, sensitivity analysis is conducted to select model parameters that have significant impact on the model predictions for the displacement, and then a modular Bayesian approach is performed to calibrate the selected model parameters using experimental displacement data from lab tests under the “pressure only” loading conditions. Challenge II is addressed using a Bayesian model calibration and bias correction approach. For improving predictions of displacement under “pressure plus liquid” loading conditions, a spatial random process (SRP) based model bias correction approach is applied to develop a refined predictive model using experimental displacement data from field tests. For challenge III, the underlying relationship between stress and displacement is identified by training a machine learning model on the simulation data generated from the supplied tank model. Final predictions of stress are made via the machine learning model and using predictions of displacements from the bias-corrected predictive model. The proposed approach not only allows the quantification of multiple sources of uncertainty and errors in the given computer models, but also is able to combine multiple sources of information to improve model performance predictions in untested domains.

P1060USING ARTIFICIAL INTELLIGENCE TO PREDICT HOME THERAPY CANDIDATES

2020 ◽  
Vol 35 (Supplement_3) ◽  
Author(s):  
Jerry Yu ◽  
Andrew Long ◽  
Maria Hanson ◽  
Aleetha Ellis ◽  
Michael Macarthur ◽  
...  
Keyword(s):  
Machine Learning ◽  
Predictive Model ◽  
Feedback Loop ◽  
Area Under The Curve ◽  
Patient Characteristics ◽  
Training Data ◽  
Quality Improvement Initiative ◽  
Home Therapy ◽  
Test Dataset ◽  
Machine Learning Model

Abstract Background and Aims There are many benefits for performing dialysis at home including more flexibility and more frequent treatments. A possible barrier to election of home therapy (HT) by in-center patients is a lack of adequate HT education. To aid efficient education efforts, a predictive model was developed to help identify patients who are more likely to switch from in-center and succeed on HT. Method We developed a model using machine learning to predict which patients who are treated in-center without prior HT history are most likely to switch to HT in the next 90 days and stay on HT for at least 90 days. Training data was extracted from 2016–2019 for approximately 300,000 patients. We randomly sampled one in-center treatment date per patient and determined if the patient would switch and succeed on HT. The input features consisted of treatment vitals, laboratories, absence history, comprehensive assessments, facility information, county-level housing, and patient characteristics. Patients were excluded if they had less than 30 days on dialysis due to lack of data. A machine learning model (XGBoost classifier) was deployed monthly in a pilot with a team of HT educators to investigate the model’s utility for identifying HT candidates. Results There were approximately 1,200 patients starting a home therapy per month in a large dialysis provider, with approximately one-third being in-center patients. The prevalence of switching and succeeding to HT in this population was 2.54%. The predictive model achieved an area under the curve of 0.87, sensitivity of 0.77, and a specificity of 0.80 on a hold-out test dataset. The pilot was successfully executed for several months and two major lessons were learned: 1) some patients who reappeared on each month’s list should be removed from the list after expressing no interest in HT, and 2) a data collection mechanism should be put in place to capture the reasons why patients are not interested in HT. Conclusion This quality-improvement initiative demonstrates that predictive modeling can be used to identify patients likely to switch and succeed on home therapy. Integration of the model in existing workflows requires creating a feedback loop which can help improve future worklists.

Updating Predictive Models: Calibration, Bias Correction and Identifiability

2010 ◽  
Author(s):  
Paul D. Arendt ◽  
Wei Chen ◽  
Daniel W. Apley
Keyword(s):  
Computer Model ◽  
Bayesian Approach ◽  
Bias Correction ◽  
Process Model ◽  
Model Updating ◽  
Future Research ◽  
Model Parameters ◽  
Sources Of Uncertainty ◽  
Uncertain Model ◽  
Many Sources

Model updating, which utilizes mathematical means to combine model simulations with physical observations for improving model predictions, has been viewed as an integral part of a model validation process. While calibration is often used to “tune” uncertain model parameters, bias-correction has been used to capture model inadequacy due to a lack of knowledge of the physics of a problem. While both sources of uncertainty co-exist, these two techniques are often implemented separately in model updating. This paper examines existing approaches to model updating and presents a modular Bayesian approach as a comprehensive framework that accounts for many sources of uncertainty in a typical model updating process and provides stochastic predictions for the purpose of design. In addition to the uncertainty in the computer model parameters and the computer model itself, this framework accounts for the experimental uncertainty and the uncertainty due to the lack of data in both computer simulations and physical experiments using the Gaussian process model. Several challenges are apparent in the implementation of the modular Bayesian approach. We argue that distinguishing between uncertain model parameters (calibration) and systematic inadequacies (bias correction) is often quite challenging due to an identifiability issue. We present several explanations and examples of this issue and bring up the needs of future research in distinguishing between the two sources of uncertainty.

scholarly journals Machine Learning Model for GSM BSC Control Plane Units

2019 ◽  
Vol 8 (6S) ◽  
pp. 219-223
Keyword(s):  
Machine Learning ◽  
Back Propagation ◽  
Back Propagation Neural Network ◽  
Model Parameters ◽  
Large Set ◽  
Data Set ◽  
Wide Acceptance ◽  
Machine Learning Model ◽  
Machine Learning Approach ◽  
Accuracy Of Prediction

At maximum traffic intensity i.e. during the busy hour, the GSM BSC signalling units (BSU) measured CPU load will be at its peak. The BSUs CPU load is a function of the number of transceivers (TRXs) mapped to it and hence the volume of offered traffic being handled by the unit. The unit CPU load is also a function of the nature of the offered load, i.e. with the same volume of offered load, the CPU load with the nominal traffic profile would be different as compared to some other arbitrary traffic profile. To manage future traffic growth, a model to estimate the BSU unit CPU load is an essential need. In recent times, using Machine Learning (ML) to develop such a model is an approach that has gained wide acceptance. Since, the estimation of CPU load is difficult as it depends on large set of parameters, machine learning approach is more scalable. In this paper, we describe a back-propagation neural network model that was developed to estimate the BSU unit CPU load. We describe the model parameters and choices and implementation architecture, and estimate its accuracy of prediction, based on an evaluation data set. We also discuss alternative ML architectures and compare their relative prediction accuracies, to the primary ML model

A machine learning approach to the observation operator for satellite radiance data assimilation

2021 ◽  
Author(s):  
Jianyu Liang ◽  
Koji Terasaki ◽  
Takemasa Miyoshi
Keyword(s):  
Machine Learning ◽  
Data Assimilation ◽  
Bias Correction ◽  
Reference System ◽  
Weather Prediction ◽  
Learning Model ◽  
Correction Procedure ◽  
Observation Operator ◽  
Machine Learning Model ◽  
Physically Based

<p>The ‘observation operator’ is essential in data assimilation (DA) to derive the model equivalent of the observations from the model variables. For satellite radiance observations, it is usually based on complex radiative transfer model (RTM) with a bias correction procedure. Therefore, it usually takes time to start using new satellite data after launching the satellites. Here we take advantage of the recent fast development of machine learning (ML) which is good at finding the complex relationships within data. ML can potentially be used as the ‘observation operator’ to reveal the relationships between the model variables and the observations without knowing their physical relationships. In this study, we test with the numerical weather prediction system composed of the Nonhydrostatic Icosahedral Atmospheric Model (NICAM) and the Local Ensemble Transform Kalman Filter (LETKF). We focus on the satellite microwave brightness temperature (BT) from the Advanced Microwave Sounding Unit-A (AMSU-A). Conventional observations and AMSU-A data were assimilated every 6 hours. The reference DA system employed the observation operator based on the RTTOV and an online bias correction method.</p><p>We used this reference system to generate 1-month data to train the machine learning model. Since the reference system includes running a physically-based RTM, we implicitly used the information from RTM for training the ML model in this study, although in our future research we will explore methods without the use of RTM. The machine learning model is artificial neural networks with 5 fully connected layers. The input of the ML model includes the NICAM model variables and predictors for bias correction, and the output of the ML model is the corresponding satellite BT in 3 channels from 5 satellites. Next, we ran the DA cycle for the same month the following year to test the performance of the ML model. Two experiments were conducted. The control experiment (CTRL) was performed with the reference system. In the test experiment (TEST), the ML model was used as the observation operator and there is no separate bias correction procedure since the training includes biased differences between the model and observation. The results showed no significant bias of the simulated BT by the ML model. Using the ECMWF global atmospheric reanalysis (ERA-interim) as a benchmark to evaluate the analysis accuracy, the global-mean RMSE, bias, and ensemble spread for temperature in TEST are 2% higher, 4% higher, and 1% lower respectively than those in CTRL. The result is encouraging since our ML can emulate the RTM. The limitation of our study is that we rely on the physically-based RTM in the reference DA system, which is used for training the ML model. This is the first result and still preliminary. We are currently considering other methods to train the ML model without using the RTM at all.</p>

A Sequential Calibration and Validation Framework for Model Parameter Updating and Bias Correction

2020 ◽  
Author(s):  
Chen Jiang ◽  
Yixuan Liu ◽  
Zhen Hu ◽  
Zissimos P. Mourelatos ◽  
David Gorsich ◽  
...  
Keyword(s):  
Model Validation ◽  
Bias Correction ◽  
Model Parameters ◽  
Model Bias ◽  
Bayesian Calibration ◽  
Prior Distributions ◽  
Model Parameter ◽  
Calibration And Validation ◽  
Validation Framework ◽  
Uncertain Model

Abstract Model parameter updating and bias correction plays an essential role in improving the validity of Modeling and Simulation (M&S) in engineering design and analysis. However, it is observed that the existing methods may either be misled by potentially wrong information if the computer model cannot adequately capture the underlying true physics, or be affected by the prior distributions of the unknown model parameters. In this paper, a sequential model calibration and validation (SeCAV) framework is proposed to improve the efficacy of both model parameter updating and model bias correction, where the model validation and Bayesian calibration are implemented in a sequential manner. In each iteration, the model validation assessment is employed as a filter to select the best experimental data for Bayesian calibration, and to update the prior distributions of uncertain model parameters for the next iteration. The calibrated parameters are then integrated with model bias correction to improve the prediction accuracy of the M&S. A mathematical example is employed to demonstrate the advantages of the SeCAV method.

scholarly journals Privacy-Preserving Gaussian Process Regression – A Modular Approach to the Application of Homomorphic Encryption

2020 ◽  
Vol 34 (04) ◽  
pp. 3866-3873
Author(s):  
Peter Fenner ◽  
Edward Pyzer-Knapp
Keyword(s):  
Machine Learning ◽  
Gaussian Process ◽  
Data Privacy ◽  
Homomorphic Encryption ◽  
Gaussian Process Regression ◽  
Machine Learning Algorithms ◽  
Modular Approach ◽  
Multiple Sources ◽  
Machine Learning Model ◽  
Evaluation Of Data

Much of machine learning relies on the use of large amounts of data to train models to make predictions. When this data comes from multiple sources, for example when evaluation of data against a machine learning model is offered as a service, there can be privacy issues and legal concerns over the sharing of data. Fully homomorphic encryption (FHE) allows data to be computed on whilst encrypted, which can provide a solution to the problem of data privacy. However, FHE is both slow and restrictive, so existing algorithms must be manipulated to make them work efficiently under the FHE paradigm. Some commonly used machine learning algorithms, such as Gaussian process regression, are poorly suited to FHE and cannot be manipulated to work both efficiently and accurately. In this paper, we show that a modular approach, which applies FHE to only the sensitive steps of a workflow that need protection, allows one party to make predictions on their data using a Gaussian process regression model built from another party's data, without either party gaining access to the other's data, in a way which is both accurate and efficient. This construction is, to our knowledge, the first example of an effectively encrypted Gaussian process.

scholarly journals Machine learning for observation bias correction with application to dust storm data assimilation

2019 ◽  
Vol 19 (15) ◽  
pp. 10009-10026 ◽  
Author(s):  
Jianbing Jin ◽  
Hai Xiang Lin ◽  
Arjo Segers ◽  
Yu Xie ◽  
Arnold Heemink
Keyword(s):  
Machine Learning ◽  
Data Assimilation ◽  
Dust Storm ◽  
Bias Correction ◽  
Dust Emission ◽  
Learning Model ◽  
Life Cycles ◽  
Dust Aerosols ◽  
Machine Learning Model ◽  
Observation Bias

Abstract. Data assimilation algorithms rely on a basic assumption of an unbiased observation error. However, the presence of inconsistent measurements with nontrivial biases or inseparable baselines is unavoidable in practice. Assimilation analysis might diverge from reality since the data assimilation itself cannot distinguish whether the differences between model simulations and observations are due to the biased observations or model deficiencies. Unfortunately, modeling of observation biases or baselines which show strong spatiotemporal variability is a challenging task. In this study, we report how data-driven machine learning can be used to perform observation bias correction for data assimilation through a real application, which is the dust emission inversion using PM10 observations. PM10 observations are considered unbiased; however, a bias correction is necessary if they are used as a proxy for dust during dust storms since they actually represent a sum of dust particles and non-dust aerosols. Two observation bias correction methods have been designed in order to use PM10 measurements as proxy for the dust storm loads under severe dust conditions. The first one is the conventional chemistry transport model (CTM) that simulates life cycles of non-dust aerosols. The other one is the machine-learning model that describes the relations between the regular PM10 and other air quality measurements. The latter is trained by learning using 2 years of historical samples. The machine-learning-based non-dust model is shown to be in better agreement with observations compared to the CTM. The dust emission inversion tests have been performed, through assimilating either the raw measurements or the bias-corrected dust observations using either the CTM or machine-learning model. The emission field, surface dust concentration, and forecast skill are evaluated. The worst case is when we directly assimilate the original observations. The forecasts driven by the a posteriori emission in this case even result in larger errors than the reference prediction. This shows the necessities of bias correction in data assimilation. The best results are obtained when using the machine-learning model for bias correction, with the existing measurements used more precisely and the resulting forecasts close to reality.

scholarly journals Uplift modeling VS conventional predictive model: A reliable machine learning model to solve employee turnover

2021 ◽  
Vol 5 (1) ◽  
Author(s):  
Davin Wijaya ◽  
Jumri Habbeyb DS ◽  
Samuelta Barus ◽  
Beriman Pasaribu ◽  
Loredana Ioana Sirbu ◽  
...  
Keyword(s):  
Machine Learning ◽  
Success Rate ◽  
Predictive Model ◽  
Employee Turnover ◽  
Performance Comparison ◽  
Uplift Modeling ◽  
Machine Learning Model ◽  
Average Accuracy ◽  
Synthetic Datasets ◽  
The Right

Employee turnover is the loss of talent in the workforce that can be costly for a company. Uplift modeling is one of the prescriptive methods in machine learning models that not only predict an outcome but also prescribe a solution. Recent studies are focusing on the conventional predictive models to predict employee turnover rather than uplift modeling. In this research, we analyze whether the uplifting model has better performance than the conventional predictive model in solving employee turnover. Performance comparison between the two methods was carried out by experimentation using two synthetic datasets and one real dataset. The results show that despite the conventional predictive model yields an average prediction accuracy of 84%; it only yields a success rate of 50% to target the right employee with a retention program on the three datasets. By contrast, the uplift model only yields an average accuracy of 67% but yields a consistent success rate of 100% in targeting the right employee with a retention program.

scholarly journals Natural springs’ protection and probabilistic risk assessment under uncertain conditions

2020 ◽  
Author(s):  
Emanuela Bianchi Janetti ◽  
Monica Riva ◽  
Alberto Guadagnini
Keyword(s):  
Conceptual Model ◽  
Risk Model ◽  
Model Parameters ◽  
System Failure ◽  
Multiple Sources ◽  
Tree Model ◽  
Field Case ◽  
Sources Of Uncertainty ◽  
Natural Springs ◽  
Probabilistic Risk

<p>We introduce, develop and test a novel Groundwater Probabilistic Risk Model, GPRM, aimed at assessing (and preventing) negative issues related to water resources management and exploitation. We apply GPRM to a highly heterogeneous regional field case, located in Northern Italy. Different risk pathways are presented formally forming a fault tree model, which enables identification of all basic events contributing to an (undesired) system failure. The latter is quantified in terms of depletion of a natural springs system representing a key feature of the considered groundwater system. The proposed GPRM allows to include the effect of multiple sources of uncertainty in our knowledge and description of the system on the evaluation of the overall probability of system failure due to different pumping schemes. In this context, we consider two probabilistic models based on different reconstruction of the aquifer geological structure. In each conceptual model, hydraulic conductivity associated with the geomaterials composing the aquifer and the boundary conditions are affected by uncertainty. Our results demonstrate that the application of GPRM to the field case allows (i) to quantify the risk associated with springs depletion due to increasing exploitation of the aquifer; (ii) to quantify how different sources of uncertainty (conceptual model uncertainty and model parameters’ uncertainty) affects this risk; (iii) to determine the optimal pumping scheme; and (iv) to identify the most vulnerable springs, where depletion first occurs.</p>

Sign in / Sign up

Export Citation Format

Share Document