scholarly journals Functional traits and community composition: multilevel models outperform community-weighted means

2017 ◽  
Author(s):  
Jesse E D Miller ◽  
Anthony Ives ◽  
Ellen Damschen
Keyword(s):  
Functional Traits ◽  
Simulation Study ◽  
Community Assembly ◽  
Fixed Effects ◽  
Multilevel Models ◽  
Type I Error ◽  
Type I ◽  
Type I Errors ◽  
Weighted Means ◽  
Inflated Type

1. Plant functional traits are increasingly being used to infer mechanisms about community assembly and predict global change impacts. Of the several approaches that are used to analyze trait-environment relationships, one of the most popular is community-weighted means (CWM), in which species trait values are averaged at the site level. Other approaches that do not require averaging are being developed, including multilevel models (MLM, also called generalized linear mixed models). However, relative strengths and weaknesses of these methods have not been extensively compared. 2. We investigated three statistical models for trait-environment associations: CWM, a MLM in which traits were not included as fixed effects (MLM1), and a MLM with traits as fixed effects (MLM2). We analyzed a real plant community dataset to investigate associations between two traits and one environmental variable. We then analyzed permutations of the dataset to investigate sources of type I errors, and performed a simulation study to compare the statistical power of the methods. 3. In the analysis of real data, CWM gave highly significant associations for both traits, while MLM1 and MLM2 did not. Using P-values derived by simulating the data using the fitted MLM2, none of the models gave significant associations, showing that CWM had inflated type I errors (false positives). In the permutation tests, MLM2 performed the best of the three approaches. MLM2 still had inflated type I error rates in some situations, but this could be corrected using bootstrapping. The simulation study showed that MLM2 always had as good or better power than CWM. These simulations also confirmed the causes of type I errors from the permutation study. 4. The MLM that includes main effects of traits (MLM2) is the best method for identifying trait-environmental association in community assembly, with better type I error control and greater power. Analyses that regress CWMs on continuous environmental variables are not reliable because they are likely to produce type I errors.

Repeated Measures Multiple Comparison Procedures: Effects of Violating Multisample Sphericity in Unbalanced Designs

1988 ◽  
Vol 13 (3) ◽  
pp. 215-226 ◽  
Author(s):  
H. J. Keselman ◽  
Joanne C. Keselman
Keyword(s):  
Repeated Measures ◽  
Type I Error ◽  
Multiple Comparison ◽  
Type I ◽  
Type I Errors ◽  
Unbalanced Designs ◽  
Bonferroni Procedure ◽  
Weighted Means ◽  
Multiple Comparison Procedures ◽  
Multisample Sphericity

Two Tukey multiple comparison procedures as well as a Bonferroni and multivariate approach were compared for their rates of Type I error and any-pairs power when multisample sphericity was not satisfied and the design was unbalanced. Pairwise comparisons of unweighted and weighted repeated measures means were computed. Results indicated that heterogenous covariance matrices in combination with unequal group sizes resulted in substantially inflated rates of Type I error for all MCPs involving comparisons of unweighted means. For tests of weighted means, both the Bonferroni and a multivariate critical value limited the number of Type I errors; however, the Bonferroni procedure provided a more powerful test, particularly when the number of repeated measures treatment levels was large.

How Low Can You Go?

Methodology ◽  
2014 ◽  
Vol 10 (1) ◽  
pp. 1-11 ◽  
Author(s):  
Bethany A. Bell ◽  
Grant B. Morgan ◽  
Jason A. Schoeneberger ◽  
Jeffrey D. Kromrey ◽  
John M. Ferron
Keyword(s):  
Confidence Interval ◽  
Sample Size ◽  
Fixed Effects ◽  
Statistical Power ◽  
Multilevel Models ◽  
Type I Error ◽  
Model Complexity ◽  
Type I ◽  
Sample Sizes ◽  
Level 1

Whereas general sample size guidelines have been suggested when estimating multilevel models, they are only generalizable to a relatively limited number of data conditions and model structures, both of which are not very feasible for the applied researcher. In an effort to expand our understanding of two-level multilevel models under less than ideal conditions, Monte Carlo methods, through SAS/IML, were used to examine model convergence rates, parameter point estimates (statistical bias), parameter interval estimates (confidence interval accuracy and precision), and both Type I error control and statistical power of tests associated with the fixed effects from linear two-level models estimated with PROC MIXED. These outcomes were analyzed as a function of: (a) level-1 sample size, (b) level-2 sample size, (c) intercept variance, (d) slope variance, (e) collinearity, and (f) model complexity. Bias was minimal across nearly all conditions simulated. The 95% confidence interval coverage and Type I error rate tended to be slightly conservative. The degree of statistical power was related to sample sizes and level of fixed effects; higher power was observed with larger sample sizes and level-1 fixed effects.

scholarly journals Monte Carlo simulation of OLS and linear mixed model inference of phenotypic effects on gene expression

PeerJ ◽  
2016 ◽  
Vol 4 ◽  
pp. e2575
Author(s):  
Jeffrey A. Walker
Keyword(s):  
Monte Carlo Simulation ◽  
Monte Carlo ◽  
Fixed Effects ◽  
Linear Mixed Model ◽  
Type I Error ◽  
Well Being ◽  
Standard Errors ◽  
Type I ◽  
Gene Set ◽  
Inflated Type

BackgroundSelf-contained tests estimate and test the association between a phenotype and mean expression level in a gene set defineda priori. Many self-contained gene set analysis methods have been developed but the performance of these methods for phenotypes that are continuous rather than discrete and with multiple nuisance covariates has not been well studied. Here, I use Monte Carlo simulation to evaluate the performance of both novel and previously published (and readily available via R) methods for inferring effects of a continuous predictor on mean expression in the presence of nuisance covariates. The motivating data are a high-profile dataset which was used to show opposing effects of hedonic and eudaimonic well-being (or happiness) on the mean expression level of a set of genes that has been correlated with social adversity (the CTRA gene set). The original analysis of these data used a linear model (GLS) of fixed effects with correlated error to infer effects ofHedoniaandEudaimoniaon mean CTRA expression.MethodsThe standardized effects ofHedoniaandEudaimoniaon CTRA gene set expression estimated by GLS were compared to estimates using multivariate (OLS) linear models and generalized estimating equation (GEE) models. The OLS estimates were tested using O’Brien’s OLS test, Anderson’s permutation ${r}_{F}^{2}$-test, two permutationF-tests (including GlobalAncova), and a rotationz-test (Roast). The GEE estimates were tested using a Wald test with robust standard errors. The performance (Type I, II, S, and M errors) of all tests was investigated using a Monte Carlo simulation of data explicitly modeled on the re-analyzed dataset.ResultsGLS estimates are inconsistent between data sets, and, in each dataset, at least one coefficient is large and highly statistically significant. By contrast, effects estimated by OLS or GEE are very small, especially relative to the standard errors. Bootstrap and permutation GLS distributions suggest that the GLS results in downward biased standard errors and inflated coefficients. The Monte Carlo simulation of error rates shows highly inflated Type I error from the GLS test and slightly inflated Type I error from the GEE test. By contrast, Type I error for all OLS tests are at the nominal level. The permutationF-tests have ∼1.9X the power of the other OLS tests. This increased power comes at a cost of high sign error (∼10%) if tested on small effects.DiscussionThe apparently replicated pattern of well-being effects on gene expression is most parsimoniously explained as “correlated noise” due to the geometry of multiple regression. The GLS for fixed effects with correlated error, or any linear mixed model for estimating fixed effects in designs with many repeated measures or outcomes, should be used cautiously because of the inflated Type I and M error. By contrast, all OLS tests perform well, and the permutationF-tests have superior performance, including moderate power for very small effects.

scholarly journals Revealing the functional traits that are linked to hidden environmental factors in community assembly

2020 ◽  
Author(s):  
Valério D. Pillar ◽  
Francesco Maria Sabatini ◽  
Ute Jandt ◽  
Sergio Camiz ◽  
Helge Bruelheide
Keyword(s):  
Environmental Factors ◽  
Functional Traits ◽  
Real World ◽  
Community Assembly ◽  
Type I Error ◽  
Environmental Filtering ◽  
Limiting Factors ◽  
Calcareous Grassland ◽  
Type I ◽  
Leaf Economics Spectrum

AbstractAimTo identify functional traits that best predict community assembly without knowing the driving environmental factors.MethodsWe propose a new method that is based on the correlation r(XY) between two matrices of potential community composition: matrix X is fuzzy-weighted by trait similarities of species, and matrix Y is derived by Beals smoothing using the probabilities of species co-occurrences. Since matrix X is based on one or more traits, r(XY) measures how well the traits used for fuzzy-weighting reflect the observed co-occurrence patterns. We developed an optimization algorithm that identifies those traits that maximize this correlation, together with an appropriate permutational test for significance. Using metacommunity data generated by a stochastic, individual-based, spatially explicit model, we assessed the type I error and the power of our method across different simulation scenarios, varying environmental filtering parameters, number of traits and trait correlation structures. We then applied the method to real-world community and trait data of dry calcareous grassland communities across Germany to identify, out of 49 traits, the combination of traits that maximizes r(XY).ResultsThe method correctly identified the relevant traits involved in the community assembly mechanisms specified in simulations. It had high power and accurate type I error and was robust against confounding aspects related to interactions between environmental factors, strength of limiting factors, and correlation among traits. In the grassland dataset, the method identified five traits that best explained community assembly. These traits reflected the size and the leaf economics spectrum, which are related to succession and resource supply, factors that may not be always measured in real-world situations.ConclusionsOur method successfully identified the relevant traits mediating community assembly driven by environmental factors which may be hidden for not being measured or accessible at the spatial or temporal scale of the study.

Assessing Person Fit With the Information Matrix Test

Methodology ◽  
2015 ◽  
Vol 11 (1) ◽  
pp. 3-12 ◽  
Author(s):  
Jochen Ranger ◽  
Jörg-Tobias Kuhn
Keyword(s):  
Simulation Study ◽  
Type I Error ◽  
Information Matrix ◽  
Small Samples ◽  
Type I ◽  
Person Fit ◽  
Power Of The Test ◽  
Order Expansion ◽  
Trait Stability ◽  
Information Matrix Test

In this manuscript, a new approach to the analysis of person fit is presented that is based on the information matrix test of White (1982) . This test can be interpreted as a test of trait stability during the measurement situation. The test follows approximately a χ2-distribution. In small samples, the approximation can be improved by a higher-order expansion. The performance of the test is explored in a simulation study. This simulation study suggests that the test adheres to the nominal Type-I error rate well, although it tends to be conservative in very short scales. The power of the test is compared to the power of four alternative tests of person fit. This comparison corroborates that the power of the information matrix test is similar to the power of the alternative tests. Advantages and areas of application of the information matrix test are discussed.

Correction: “Influence of Selection Bias on the Test Decision – A Simulation Study”

2014 ◽  
Vol 53 (05) ◽  
pp. 343-343
Keyword(s):  
Selection Bias ◽  
Simulation Study ◽  
Error Rate ◽  
Type I Error ◽  
Block Size ◽  
Error Rates ◽  
Type I ◽  
Type I Error Rate ◽  
Representation Error ◽  
Numeric Representation

We have to report marginal changes in the empirical type I error rates for the cut-offs 2/3 and 4/7 of Table 4, Table 5 and Table 6 of the paper “Influence of Selection Bias on the Test Decision – A Simulation Study” by M. Tamm, E. Cramer, L. N. Kennes, N. Heussen (Methods Inf Med 2012; 51: 138 –143). In a small number of cases the kind of representation of numeric values in SAS has resulted in wrong categorization due to a numeric representation error of differences. We corrected the simulation by using the round function of SAS in the calculation process with the same seeds as before. For Table 4 the value for the cut-off 2/3 changes from 0.180323 to 0.153494. For Table 5 the value for the cut-off 4/7 changes from 0.144729 to 0.139626 and the value for the cut-off 2/3 changes from 0.114885 to 0.101773. For Table 6 the value for the cut-off 4/7 changes from 0.125528 to 0.122144 and the value for the cut-off 2/3 changes from 0.099488 to 0.090828. The sentence on p. 141 “E.g. for block size 4 and q = 2/3 the type I error rate is 18% (Table 4).” has to be replaced by “E.g. for block size 4 and q = 2/3 the type I error rate is 15.3% (Table 4).”. There were only minor changes smaller than 0.03. These changes do not affect the interpretation of the results or our recommendations.

scholarly journals Testing for treatment effect in covariate-adaptive randomized trials with generalized linear models and omitted covariates

2021 ◽  
pp. 096228022110082
Author(s):  
Yang Li ◽  
Wei Ma ◽  
Yichen Qin ◽  
Feifang Hu
Keyword(s):  
Generalized Linear Models ◽  
Treatment Effect ◽  
Linear Models ◽  
Type I Error ◽  
Type I ◽  
Test Statistic ◽  
Asymptotic Results ◽  
Adaptive Randomization ◽  
Adjustment Method ◽  
Inflated Type

Concerns have been expressed over the validity of statistical inference under covariate-adaptive randomization despite the extensive use in clinical trials. In the literature, the inferential properties under covariate-adaptive randomization have been mainly studied for continuous responses; in particular, it is well known that the usual two-sample t-test for treatment effect is typically conservative. This phenomenon of invalid tests has also been found for generalized linear models without adjusting for the covariates and are sometimes more worrisome due to inflated Type I error. The purpose of this study is to examine the unadjusted test for treatment effect under generalized linear models and covariate-adaptive randomization. For a large class of covariate-adaptive randomization methods, we obtain the asymptotic distribution of the test statistic under the null hypothesis and derive the conditions under which the test is conservative, valid, or anti-conservative. Several commonly used generalized linear models, such as logistic regression and Poisson regression, are discussed in detail. An adjustment method is also proposed to achieve a valid size based on the asymptotic results. Numerical studies confirm the theoretical findings and demonstrate the effectiveness of the proposed adjustment method.

scholarly journals Another Warning about Median Reaction Time --- Version of 11 Feb 2020

2020 ◽  
Author(s):  
Jeff Miller
Keyword(s):  
Type I Error ◽  
Reaction Times ◽  
Error Rates ◽  
Type I ◽  
Experimental Conditions ◽  
Type I Error Rates ◽  
Correction Technique ◽  
Inflated Type ◽  
The Mean ◽  
Rt Distributions

Contrary to the warning of Miller (1988), Rousselet and Wilcox (2020) argued that it is better to summarize each participant’s single-trial reaction times (RTs) in a given condition with the median than with the mean when comparing the central tendencies of RT distributions across experimental conditions. They acknowledged that median RTs can produce inflated Type I error rates when conditions differ in the number of trials tested, consistent with Miller’s warning, but they showed that the bias responsible for this error rate inflation could be eliminated with a bootstrap bias correction technique. The present simulations extend their analysis by examining the power of bias-corrected medians to detect true experimental effects and by comparing this power with the power of analyses using means and regular medians. Unfortunately, although bias-corrected medians solve the problem of inflated Type I error rates, their power is lower than that of means or regular medians in many realistic situations. In addition, even when conditions do not differ in the number of trials tested, the power of tests (e.g., t-tests) is generally lower using medians rather than means as the summary measures. Thus, the present simulations demonstrate that summary means will often provide the most powerful test for differences between conditions, and they show what aspects of the RT distributions determine the size of the power advantage for means.

scholarly journals Generalized additive models and inflated type I error rates of smoother significance tests

2011 ◽  
Vol 55 (1) ◽  
pp. 366-374 ◽  
Author(s):  
Robin L. Young ◽  
Janice Weinberg ◽  
Verónica Vieira ◽  
Al Ozonoff ◽  
Thomas F. Webster
Keyword(s):  
Type I Error ◽  
Generalized Additive Models ◽  
Error Rates ◽  
Additive Models ◽  
Type I ◽  
Significance Tests ◽  
Type I Error Rates ◽  
Inflated Type
Sign in / Sign up

Export Citation Format

Share Document