An Unbiased Model Comparison Test Using Cross-Validation

2011 ◽  
Author(s):  
Bruce A. Desmarais ◽  
Jeffrey J. Harden
Keyword(s):  
Model Comparison ◽  
Cross Validation ◽  
Comparison Test

An unbiased model comparison test using cross-validation

2013 ◽  
Vol 48 (4) ◽  
pp. 2155-2173 ◽  
Author(s):  
Bruce A. Desmarais ◽  
Jeffrey J. Harden
Keyword(s):  
Model Comparison ◽  
Cross Validation ◽  
Comparison Test

Composite likelihood model comparison test under fixed and local alternatives

Stat ◽  
2018 ◽  
Vol 7 (1) ◽  
pp. e182
Author(s):  
Yawen Xu ◽  
Xin Gao ◽  
Xiaogang Wang ◽  
Augustine Wong
Keyword(s):  
Model Comparison ◽  
Composite Likelihood ◽  
Local Alternatives ◽  
Comparison Test ◽  
Likelihood Model

scholarly journals A Parameter-Free Model Comparison Test Using Differential Algebra

Complexity ◽  
2019 ◽  
Vol 2019 ◽  
pp. 1-15
Author(s):  
Heather A. Harrington ◽  
Kenneth L. Ho ◽  
Nicolette Meshkat
Keyword(s):  
Time Course ◽  
Model Comparison ◽  
Time Series Data ◽  
Dynamic Properties ◽  
Differential Algebra ◽  
Series Data ◽  
Comparison Test ◽  
Competing Models ◽  
Model Equations ◽  
Parameter Values

We present a method for rejecting competing models from noisy time-course data that does not rely on parameter inference. First we characterize ordinary differential equation models in only measurable variables using differential-algebra elimination. This procedure gives input-output equations, which serve as invariants for time series data. We develop a model comparison test using linear algebra and statistics to reject incorrect models from their invariants. This algorithm exploits the dynamic properties that are encoded in the structure of the model equations without recourse to parameter values, and, in this sense, the approach is parameter-free. We demonstrate this method by discriminating between different models from mathematical biology.

scholarly journals New Developments in Factor Score Regression: Fit Indices and a Model Comparison Test

2019 ◽  
Vol 79 (6) ◽  
pp. 1017-1037 ◽  
Author(s):  
Ines Devlieger ◽  
Wouter Talloen ◽  
Yves Rosseel
Keyword(s):  
Structural Equation Modeling ◽  
Structural Equation ◽  
Model Comparison ◽  
Type I Error ◽  
Factor Score ◽  
Regression Coefficients ◽  
Equation Modeling ◽  
Type I ◽  
Fit Indices ◽  
Comparison Test

Factor score regression (FSR) is a popular alternative for structural equation modeling. Naively applying FSR induces bias for the estimators of the regression coefficients. Croon proposed a method to correct for this bias. Next to estimating effects without bias, interest often lies in inference of regression coefficients or in the fit of the model. In this article, we propose fit indices for FSR that can be used to inspect the model fit. We also introduce a model comparison test based on one of these newly proposed fit indices that can be used for inference of the estimators on the regression coefficients. In a simulation study we compare FSR with Croon’s corrections and structural equation modeling in terms of bias of the regression coefficients, Type I error rate and power.

scholarly journals Bayesian cross validation for gravitational-wave searches in pulsar-timing array data

2019 ◽  
Vol 487 (3) ◽  
pp. 3644-3649 ◽  
Author(s):  
Haochen Wang ◽  
Stephen R Taylor ◽  
Michele Vallisneri
Keyword(s):  
Data Analysis ◽  
Gravitational Wave ◽  
Bayesian Model ◽  
Model Comparison ◽  
Cross Validation ◽  
Training Data ◽  
Data Set ◽  
Bayesian Model Comparison ◽  
Pulsar Timing ◽  
Pulsar Timing Array

ABSTRACT Gravitational-wave data analysis demands sophisticated statistical noise models in a bid to extract highly obscured signals from data. In Bayesian model comparison, we choose among a landscape of models by comparing their marginal likelihoods. However, this computation is numerically fraught and can be sensitive to arbitrary choices in the specification of parameter priors. In Bayesian cross validation, we characterize the fit and predictive power of a model by computing the Bayesian posterior of its parameters in a training data set, and then use that posterior to compute the averaged likelihood of a different testing data set. The resulting cross-validation scores are straightforward to compute; they are insensitive to prior tuning; and they penalize unnecessarily complex models that overfit the training data at the expense of predictive performance. In this article, we discuss cross validation in the context of pulsar-timing-array data analysis, and we exemplify its application to simulated pulsar data (where it successfully selects the correct spectral index of a stochastic gravitational-wave background), and to a pulsar data set from the NANOGrav 11-yr release (where it convincingly favours a model that represents a transient feature in the interstellar medium). We argue that cross validation offers a promising alternative to Bayesian model comparison, and we discuss its use for gravitational-wave detection, by selecting or refuting models that include a gravitational-wave component.

scholarly journals Rejoinder: More Limitations of Bayesian Leave-One-Out Cross-Validation

2018 ◽  
Author(s):  
Quentin Frederik Gronau ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Parameter Estimation ◽  
Model Selection ◽  
Model Comparison ◽  
Cross Validation ◽  
Mathematical Psychology ◽  
Bayes Rule ◽  
Epistemic Goal ◽  
Leave One Out

We recently discussed several limitations of Bayesian leave-one-out cross-validation (LOO) for model selection. Our contribution attracted three thought-provoking commentaries. In this rejoinder, we address each of the commentaries and identify several additional limitations of LOO-based methods such as Bayesian stacking. We focus on differences between LOO-based methods versus approaches that consistently use Bayes' rule for both parameter estimation and model comparison. We conclude that LOO-based methods do not align satisfactorily with the epistemic goal of mathematical psychology.

scholarly journals mcp: An R Package for Regression With Multiple Change Points

2020 ◽  
Author(s):  
Jonas Kristoffer Lindeløv
Keyword(s):  
Change Point ◽  
General Model ◽  
Regression Models ◽  
Model Comparison ◽  
Cross Validation ◽  
Density Ratio ◽  
R Package ◽  
Bayesian Regression ◽  
Bayes Factors ◽  
Change Points

The R package mcp does flexible and informed Bayesian regression with change points. mcp can infer the location of changes between regression models on means, variances, autocorrelation structure, and any combination of these. Prior and posterior samples and summaries are returned for all parameters and a rich set of plotting options is available. Bayes Factors can be computed via Savage-Dickey density ratio and posterior contrasts. Cross-validation can be used for more general model comparison. mcp ships with sensible defaults, including priors, but the user can override them to get finer control of the models and outputs. The strengths and limitations of mcp are discussed in relation to existing change point packages in R.

scholarly journals modeLLtest: An R Package for Unbiased Model Comparison using Cross Validation

2019 ◽  
Vol 4 (41) ◽  
pp. 1542
Author(s):  
Shana Scogin ◽  
Sarah Petersen ◽  
Jeffrey Harden ◽  
Bruce Desmarais
Keyword(s):  
Model Comparison ◽  
Cross Validation ◽  
R Package

scholarly journals Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models

2020 ◽  
Author(s):  
Paul-Christian Bürkner ◽  
Jonah Gabry ◽  
Aki Vehtari
Keyword(s):  
Spatial Statistics ◽  
Model Comparison ◽  
Cross Validation ◽  
Predictive Accuracy ◽  
Observation Model ◽  
Multivariate Normal ◽  
Temporal And Spatial ◽  
Leave One Out

AbstractCross-validation can be used to measure a model’s predictive accuracy for the purpose of model comparison, averaging, or selection. Standard leave-one-out cross-validation (LOO-CV) requires that the observation model can be factorized into simple terms, but a lot of important models in temporal and spatial statistics do not have this property or are inefficient or unstable when forced into a factorized form. We derive how to efficiently compute and validate both exact and approximate LOO-CV for any Bayesian non-factorized model with a multivariate normal or Student-$$t$$ t distribution on the outcome values. We demonstrate the method using lagged simultaneously autoregressive (SAR) models as a case study.

Identifying suboptimalities with factorial model comparison

2018 ◽  
Vol 41 ◽  
Author(s):  
Wei Ji Ma
Keyword(s):  
Experimental Design ◽  
Model Comparison ◽  
Process Models ◽  
Multiple Forms ◽  
Factorial Experimental Design ◽  
Factorial Model ◽  
Multiple Components ◽  
The Many

AbstractGiven the many types of suboptimality in perception, I ask how one should test for multiple forms of suboptimality at the same time – or, more generally, how one should compare process models that can differ in any or all of the multiple components. In analogy to factorial experimental design, I advocate for factorial model comparison.

Sign in / Sign up

Export Citation Format

Share Document