A Methodological Approach to Assessment and Reporting of the Model Adequacy in Simulation Studies Abstract

Assessment and Reporting of Model Adequacy is an important step in the simulation modelling process. It stipulates the level of precision and accuracy, which are important features of the model predictions. In an academic research activity, an important step for model development is the process of the identification or accepting whether the model is wrong. The evaluation of the adequacy of developed models is not possible through a single statistical test. This paper delineates a technique to implement model adequacy. A live case is demonstrated on the proposed methodology by evaluating a simulation model which was designed by us to simulate a well-established mathematical model. A step by step methodological approach is delineated in this paper along with a case study of investigation of a simulation model with a mathematical model is used to demonstrate this methodology. The paper concludes with an Algorithm and a flow chart for performing model adequacy for assessing the adequacy of the developed model with existing models.

A Methodological Approach to Assessment and Reporting of the Model Adequacy in Simulation Studies

Assessment and Reporting of Model Adequacy is an important step in the simulation modelling process. It stipulates the level of precision and accuracy, which are important features of the model predictions. In an academic research activity, an important step for model development is the process of the identification or accepting whether the model is wrong. The evaluation of the adequacy of developed models is not possible through a single statistical test. This paper delineates a technique to implement model adequacy. A live case is demonstrated on the proposed methodology by evaluating a simulation model which was designed by us to simulate a well-established mathematical model. A step by step methodological approach is delineated in this paper along with a case study of investigation of a simulation model with a mathematical model is used to demonstrate this methodology. The paper concludes with an Algorithm and a flow chart for performing model adequacy for assessing the adequacy of the developed model with existing models.

The Shape of Trees – Limits of Current Diversification Models

To investigate how biodiversity arose, the field of macroevolution largely relies on model-based approaches to estimate rates of diversification and what factors influence them. The number of available models is rising steadily, facilitating the modeling of an increasing number of possible diversification dynamics, and multiple hypotheses relating to what fueled or stifled lineage accumulation within groups of organisms. However, growing concerns about unchecked biases and limitations in the employed models suggest the need for rigorous validation of methods used to infer. Here, we address two points: the practical use of model adequacy testing, and what model adequacy can tell us about the overall state of diversification models. Using a large set of empirical phylogenies, and a new approach to test models using aspects of tree shape, we test how a set of staple models performs with regards to adequacy. Patterns of adequacy are described across trees and models and causes for inadequacy – particularly if all models are inadequate – are explored. The findings make clear that overall, only few empirical phylogenies cannot be described by at least one model. However, finding that the best fitting of a set of models might not necessarily be adequate makes clear that adequacy testing should become a step in the standard procedures for diversification studies.

BoskR – Testing Adequacy of Diversification Models Using Tree Shape

The study of diversification largely relies on model-based approaches, estimating rates of speciation and extinction from phylogenetic trees. While a plethora of different models exist – all with different features, strengths and weaknesses – there is increasing concern about the reliability of the inference we gain from them. Apart from simply finding the model with the best fit for the data, we should find ways to assess a model’s suitability to describe the data in an absolute sense. The R package BoskR implements a simple way of judging a model’s adequacy for a given phylogeny using metrics for tree shape, assuming that a model is inadequate for a phylogeny if it produces trees that are consistently dissimilar in shape from the tree that should be analyzed. Tree shape is assessed via metrics derived from the tree’s modified graph Laplacian spectrum, as provided by RPANDA. We exemplify the use of the method using simulated and empirical example phylogenies. BoskR was mostly able to correctly distinguish trees simulated under clearly different models and revealed that not all models are adequate for the empirical example trees. We believe the metrics of tree shape to be an intuitive and relevant means of assessing diversification model adequacy. Furthermore, by implementing the approach in an openly available R package, we enable and encourage researchers to adopt adequacy testing into their workflow.

Model Adequacy Tests for Likelihood Models of Chromosome-Number Evolution

SummaryChromosome number is a central feature of eukaryote genomes. Deciphering patterns of chromosome-number change along a phylogeny is central to the inference of whole genome duplications and ancestral chromosome numbers. ChromEvol is a probabilistic inference tool that allows the evaluation of several models of chromosome-number evolution and their fit to the data. However, fitting a model does not necessarily mean that the model describes the empirical data adequately. This vulnerability may lead to incorrect conclusions when model assumptions are not met by real data.Here, we present a model adequacy test for likelihood models of chromosome-number evolution. The procedure allows to determine whether the model can generate data with similar characteristics as those found in the observed ones.We demonstrate that using inadequate models can lead to inflated errors in several inference tasks. Applying the developed method to 200 angiosperm genera, we find that in many of these, the best-fitted model provides poor fit to the data. The inadequacy rate increases in large clades or in those in which hybridizations are present.The developed model adequacy test can help researchers to identify phylogenies whose underlying evolutionary patterns deviate substantially from current modelling assumptions and should guide future methods developments.