scholarly journals Reliability and reproducibility in computational science: implementing validation, verification and uncertainty quantification in silico

Author(s):  
Peter V. Coveney ◽  
Derek Groen ◽  
Alfons G. Hoekstra
Author(s):  
Odd Erik Gundersen

Reproducibility is a confused terminology. In this paper, I take a fundamental view on reproducibility rooted in the scientific method. The scientific method is analysed and characterized in order to develop the terminology required to define reproducibility. Furthermore, the literature on reproducibility and replication is surveyed, and experiments are modelled as tasks and problem solving methods. Machine learning is used to exemplify the described approach. Based on the analysis, reproducibility is defined and three different degrees of reproducibility as well as four types of reproducibility are specified. This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.


Author(s):  
M. S. Krafczyk ◽  
A. Shi ◽  
A. Bhaskar ◽  
D. Marinov ◽  
V. Stodden

We carry out efforts to reproduce computational results for seven published articles and identify barriers to computational reproducibility. We then derive three principles to guide the practice and dissemination of reproducible computational research: (i) Provide transparency regarding how computational results are produced; (ii) When writing and releasing research software, aim for ease of (re-)executability; (iii) Make any code upon which the results rely as deterministic as possible. We then exemplify these three principles with 12 specific guidelines for their implementation in practice. We illustrate the three principles of reproducible research with a series of vignettes from our experimental reproducibility work. We define a novel Reproduction Package , a formalism that specifies a structured way to share computational research artifacts that implements the guidelines generated from our reproduction efforts to allow others to build, reproduce and extend computational science. We make our reproduction efforts in this paper publicly available as exemplar Reproduction Packages . This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.


Author(s):  
D. Ye ◽  
L. Veen ◽  
A. Nikishova ◽  
J. Lakhlili ◽  
W. Edeling ◽  
...  

Uncertainty quantification (UQ) is a key component when using computational models that involve uncertainties, e.g. in decision-making scenarios. In this work, we present uncertainty quantification patterns (UQPs) that are designed to support the analysis of uncertainty in coupled multi-scale and multi-domain applications. UQPs provide the basic building blocks to create tailored UQ for multiscale models. The UQPs are implemented as generic templates, which can then be customized and aggregated to create a dedicated UQ procedure for multiscale applications. We present the implementation of the UQPs with multiscale coupling toolkit Multiscale Coupling Library and Environment 3. Potential speed-up for UQPs has been derived as well. As a proof of concept, two examples of multiscale applications using UQPs are presented. This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.


Author(s):  
Jeff Bodner ◽  
Vikas Kaul

Abstract The rising costs of clinical trials for medical devices in recent years has led to an increased interest in so-called in silico clinical trials, where simulation results are used to supplement or to replace those obtained from human patients. Here we present a framework for executing such a trial. This framework relies heavily on ideas already developed for model verification, validation, and uncertainty quantification. The framework uses results from an initial cohort of human patients as model validation data, recognizing that the best model credibility evidence usually comes from real patients. The validation exercise leads to an assessment of the model’s suitability based on pre-defined acceptance criteria. If the model meets these criteria, then no additional human patients are required and the study endpoints that can be addressed using the model are met using the simulation results. Conversely, if the model is found to be inadequate, it is abandoned, and the clinical study continues using only human patients in a second cohort. Compared to other frameworks described in the literature based on Bayesian methods, this approach follows a strict model build-validate-predict structure. It can handle epistemic uncertainties in the model inputs, which is a common trait of models of biomedical systems. Another idea discussed here is that the outputs of engineering models rarely coincide with measures that are the basis for clinical endpoints. This manuscript discusses how the link between the model and clinical measure can be established during the trial.


Author(s):  
D. Groen ◽  
H. Arabnejad ◽  
V. Jancauskas ◽  
W. N. Edeling ◽  
F. Jansson ◽  
...  

We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantication (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against observational or benchmark data, and run simulations conveniently on any platform from the desktop to current multi-petascale computers. In this sequel to our paper on VECMAtk which we presented last year [ 1 ] we focus on a range of functional and performance improvements that we have introduced, cover newly introduced components, and applications examples from seven different domains such as conflict modelling and environmental sciences. We also present several implemented patterns for UQ/SA and V&V, and guide the reader through one example concerning COVID-19 modelling in detail. This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico ’.


Sign in / Sign up

Export Citation Format

Share Document