scholarly journals The Connection between Bayesian Inference and Information Theory for Model Selection, Information Gain and Experimental Design

Entropy ◽  
2019 ◽  
Vol 21 (11) ◽  
pp. 1081 ◽  
Author(s):  
Sergey Oladyshkin ◽  
Wolfgang Nowak
Keyword(s):  
Information Theory ◽  
Monte Carlo ◽  
Bayesian Inference ◽  
Experimental Design ◽  
Model Selection ◽  
Relative Entropy ◽  
Information Entropy ◽  
Bayesian Model ◽  
Information Gain ◽  
Information Criteria

We show a link between Bayesian inference and information theory that is useful for model selection, assessment of information entropy and experimental design. We align Bayesian model evidence (BME) with relative entropy and cross entropy in order to simplify computations using prior-based (Monte Carlo) or posterior-based (Markov chain Monte Carlo) BME estimates. On the one hand, we demonstrate how Bayesian model selection can profit from information theory to estimate BME values via posterior-based techniques. Hence, we use various assumptions including relations to several information criteria. On the other hand, we demonstrate how relative entropy can profit from BME to assess information entropy during Bayesian updating and to assess utility in Bayesian experimental design. Specifically, we emphasize that relative entropy can be computed avoiding unnecessary multidimensional integration from both prior and posterior-based sampling techniques. Prior-based computation does not require any assumptions, however posterior-based estimates require at least one assumption. We illustrate the performance of the discussed estimates of BME, information entropy and experiment utility using a transparent, non-linear example. The multivariate Gaussian posterior estimate includes least assumptions and shows the best performance for BME estimation, information entropy and experiment utility from posterior-based sampling.

scholarly journals Bayesian3 Active Learning for the Gaussian Process Emulator Using Information Theory

Entropy ◽  
2020 ◽  
Vol 22 (8) ◽  
pp. 890
Author(s):  
Sergey Oladyshkin ◽  
Farid Mohammadi ◽  
Ilja Kroeker ◽  
Wolfgang Nowak
Keyword(s):  
Bayesian Inference ◽  
Active Learning ◽  
Gaussian Process ◽  
Relative Entropy ◽  
Bayesian Model ◽  
Information Gain ◽  
Measurement Data ◽  
Superior Performance ◽  
Evidence Based ◽  
Model Evidence

Gaussian process emulators (GPE) are a machine learning approach that replicates computational demanding models using training runs of that model. Constructing such a surrogate is very challenging and, in the context of Bayesian inference, the training runs should be well invested. The current paper offers a fully Bayesian view on GPEs for Bayesian inference accompanied by Bayesian active learning (BAL). We introduce three BAL strategies that adaptively identify training sets for the GPE using information-theoretic arguments. The first strategy relies on Bayesian model evidence that indicates the GPE’s quality of matching the measurement data, the second strategy is based on relative entropy that indicates the relative information gain for the GPE, and the third is founded on information entropy that indicates the missing information in the GPE. We illustrate the performance of our three strategies using analytical- and carbon-dioxide benchmarks. The paper shows evidence of convergence against a reference solution and demonstrates quantification of post-calibration uncertainty by comparing the introduced three strategies. We conclude that Bayesian model evidence-based and relative entropy-based strategies outperform the entropy-based strategy because the latter can be misleading during the BAL. The relative entropy-based strategy demonstrates superior performance to the Bayesian model evidence-based strategy.

scholarly journals Adaptive Estimation for Epidemic Renewal and Phylogenetic Skyline Models

2020 ◽  
Vol 69 (6) ◽  
pp. 1163-1179 ◽  
Author(s):  
Kris V Parag ◽  
Christl A Donnelly
Keyword(s):  
Information Theory ◽  
Model Selection ◽  
Adaptive Estimation ◽  
Minimum Description Length ◽  
Target Population ◽  
Theory Model ◽  
Information Criteria ◽  
Classification Problems ◽  
Effective Population ◽  
Incident Cases

Abstract Estimating temporal changes in a target population from phylogenetic or count data is an important problem in ecology and epidemiology. Reliable estimates can provide key insights into the climatic and biological drivers influencing the diversity or structure of that population and evidence hypotheses concerning its future growth or decline. In infectious disease applications, the individuals infected across an epidemic form the target population. The renewal model estimates the effective reproduction number, R, of the epidemic from counts of observed incident cases. The skyline model infers the effective population size, N, underlying a phylogeny of sequences sampled from that epidemic. Practically, R measures ongoing epidemic growth while N informs on historical caseload. While both models solve distinct problems, the reliability of their estimates depends on p-dimensional piecewise-constant functions. If p is misspecified, the model might underfit significant changes or overfit noise and promote a spurious understanding of the epidemic, which might misguide intervention policies or misinform forecasts. Surprisingly, no transparent yet principled approach for optimizing p exists. Usually, p is heuristically set, or obscurely controlled via complex algorithms. We present a computable and interpretable p-selection method based on the minimum description length (MDL) formalism of information theory. Unlike many standard model selection techniques, MDL accounts for the additional statistical complexity induced by how parameters interact. As a result, our method optimizes p so that R and N estimates properly and meaningfully adapt to available data. It also outperforms comparable Akaike and Bayesian information criteria on several classification problems, given minimal knowledge of the parameter space, and exposes statistical similarities among renewal, skyline, and other models in biology. Rigorous and interpretable model selection is necessary if trustworthy and justifiable conclusions are to be drawn from piecewise models. [Coalescent processes; epidemiology; information theory; model selection; phylodynamics; renewal models; skyline plots]

scholarly journals Accelerated Bayesian inference of gene expression models from snapshots of single-cell transcripts

2018 ◽  
Author(s):  
Yen Ting Lin ◽  
Nicolas E. Buchler
Keyword(s):  
Gene Expression ◽  
Monte Carlo ◽  
Bayesian Inference ◽  
Model Selection ◽  
Single Cell ◽  
Kinetic Parameters ◽  
Information Criterion ◽  
Time Dependent ◽  
Data Sets ◽  
Bayesian Evidence

Understanding how stochastic gene expression is regulated in biological systems using snapshots of single-cell transcripts requires state-of-the-art methods of computational analysis and statistical inference. A Bayesian approach to statistical inference is the most complete method for model selection and uncertainty quantification of kinetic parameters from single-cell data. This approach is impractical because current numerical algorithms are too slow to handle typical models of gene expression. To solve this problem, we first show that time-dependent mRNA distributions of discrete-state models of gene expression are dynamic Poisson mixtures, whose mixing kernels are characterized by a piece-wise deterministic Markov process. We combined this analytical result with a kinetic Monte Carlo algorithm to create a hybrid numerical method that accelerates the calculation of time-dependent mRNA distributions by 1000-fold compared to current methods. We then integrated the hybrid algorithm into an existing Monte Carlo sampler to estimate the Bayesian posterior distribution of many different, competing models in a reasonable amount of time. We validated our method of accelerated Bayesian inference on several synthetic data sets. Our results show that kinetic parameters can be reasonably constrained for modestly sampled data sets, if the model is known a priori. If the model is unknown,the Bayesian evidence can be used to rigorously quantify the likelihood of a model relative to other models from the data. We demonstrate that Bayesian evidence selects the true model and outperforms approximate metrics, e.g., Bayesian Information Criterion (BIC) or Akaike Information Criterion (AIC), often used for model selection.

scholarly journals Bayesian Model Selection in Fisheries Management and Ecology

2019 ◽  
Vol 10 (2) ◽  
pp. 691-707
Author(s):  
Jason C. Doll ◽  
Stephen J. Jacquemin
Keyword(s):  
Bayesian Inference ◽  
Model Selection ◽  
Fisheries Management ◽  
Bayesian Model ◽  
Bayes Factor ◽  
Evaluation Process ◽  
Information Criterion ◽  
Bayesian Model Selection ◽  
Model Parameters ◽  
Inference Methods

Abstract Researchers often test ecological hypotheses relating to a myriad of questions ranging from assemblage structure, population dynamics, demography, abundance, growth rate, and more using mathematical models that explain trends in data. To aid in the evaluation process when faced with competing hypotheses, we employ statistical methods to evaluate the validity of these multiple hypotheses with the goal of deriving the most robust conclusions possible. In fisheries management and ecology, frequentist methodologies have largely dominated this approach. However, in recent years, researchers have increasingly used Bayesian inference methods to estimate model parameters. Our aim with this perspective is to provide the practicing fisheries ecologist with an accessible introduction to Bayesian model selection. Here we discuss Bayesian inference methods for model selection in the context of fisheries management and ecology with empirical examples to guide researchers in the use of these methods. In this perspective we discuss three methods for selecting among competing models. For comparing two models we discuss Bayes factor and for more complex models we discuss Watanabe–Akaike information criterion and leave-one-out cross-validation. We also describe what kinds of information to report when conducting Bayesian inference. We conclude this review with a discussion of final thoughts about these model selection techniques.

Simplicity and Model Selection

2019 ◽  
pp. 260-286
Author(s):  
Jan Sprenger ◽  
Stephan Hartmann
Keyword(s):  
Bayesian Inference ◽  
Model Selection ◽  
Bayesian Model ◽  
Scientific Theory ◽  
Predictive Accuracy ◽  
Practical Applications ◽  
Philosophical Foundations ◽  
Selection Strategies ◽  
Statistical Model Selection ◽  
Good Scientific Theory

Is simplicity a virtue of a good scientific theory, and are simpler theories more likely to be true or predictively successful? If so, how much should simplicity count vis-à-vis predictive accuracy? We address this question using Bayesian inference, focusing on the context of statistical model selection and an interpretation of simplicity via the degree of freedoms of a model. We rebut claims to prove the epistemic value of simplicity by means of showing its particular role in Bayesian model selection strategies (e.g., the BIC or the MML). Instead, we show that Bayesian inference in the context of model selection is usually done in a philosophically eclectic, instrumental fashion that is more tuned to practical applications than to philosophical foundations. Thus, these techniques cannot justify a particular “appropriate weight of simplicity in model selection”.

Sign in / Sign up

Export Citation Format

Share Document