scholarly journals p ‐Curve analysis of the Taylor Aggression Paradigm: Estimating evidentiary value and statistical power across 50 years of research

2020 ◽  
Author(s):  
Samuel J. West ◽  
Courtland S. Hyatt ◽  
Joshua D. Miller ◽  
David S. Chester
Keyword(s):  
Statistical Power ◽  
Curve Analysis ◽  
Taylor Aggression Paradigm

scholarly journals p-Curve Analysis of the Taylor Aggression Paradigm: Estimating Evidentiary Value and Statistical Power Across 50 Years of Research

2020 ◽  
Author(s):  
Samuel James West ◽  
Courtland Hyatt ◽  
Josh Miller ◽  
David Chester
Keyword(s):  
Statistical Power ◽  
Curve Analysis ◽  
Effect Sizes ◽  
Questionable Research Practices ◽  
Psychological Measures ◽  
Taylor Aggression Paradigm ◽  
Research Paradigms ◽  
Main Effects ◽  
Overall Reliability

The overall reliability or evidentiary value of any body of literature is established in part by ruling out publication bias for any observed effects. Questionable research practices have potentially undermined the evidentiary value of commonly-used research paradigms in psychological science. Subsequently, the evidentiary value of these common methodologies remains uncertain. To quantify the severity of these issues in the literature, we selected the Taylor Aggression Paradigm (TAP) as a case study and submitted 170 hypothesis-tests spanning over 50 years of research to a preregistered p-curve analysis. The TAP literature (N = 24,685) demonstrated significant evidentiary value but yielded a small average effect size (d = .29) and inadequate power (38%). Main effects demonstrated greater evidentiary value, power, and effect sizes than interactions. Studies that tested the effects of measured traits did not differ in evidentiary value or power to those that tested the effects of experimentally-manipulated states. Exploratory analyses revealed that evidentiary value, statistical power, and effect sizes have improved over time. We provide recommendations for researchers who seek to maximize the evidentiary value of their psychological measures.

Assessing the evidence of perspective taking on stereotyping and negative evaluations: A p-curve analysis

2020 ◽  
pp. 136843022095708
Author(s):  
Qian Huang ◽  
Wei Peng ◽  
Jazmyne V. Simmons
Keyword(s):  
Social Science ◽  
Attitude Change ◽  
Perspective Taking ◽  
Statistical Power ◽  
Experimental Studies ◽  
Curve Analysis ◽  
Evidential Value ◽  
Intergroup Attitude ◽  
Science Disciplines

Perspective taking is conceptualized as the ability to consider or adopt the perspective of another individual who is perceived to be in need; it has shown mixed results in stereotype reduction and intergroup attitude change across many social science disciplines. The inconsistent results raise concerns about the robustness of the perspective-taking phenomenon. The present study uses p-curve analysis to examine whether evidential value existed among two sets of published experimental studies where perspective taking was operationalized in two different paradigms. Despite low statistical power, we found that both sets of studies revealed some evidential value of the effects of perspective taking. The theoretical and methodological implications of perspective-taking studies are discussed as well.

Power-Enhanced Funnel Plots for Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 43-49 ◽  
Author(s):  
Michael Kossmeier ◽  
Ulrich S. Tran ◽  
Martin Voracek
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Relevant Information ◽  
Selective Reporting ◽  
Graphical Display ◽  
Funnel Plot ◽  
Evidential Value ◽  
Graphical Displays ◽  
Funnel Plots ◽  
Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

How to Detect Publication Bias in Psychological Research

2019 ◽  
Vol 227 (4) ◽  
pp. 261-279 ◽  
Author(s):  
Frank Renkewitz ◽  
Melanie Keiner
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Statistical Power ◽  
Type I Error ◽  
Psychological Research ◽  
Type I ◽  
True Effect Size ◽  
Questionable Research Practices ◽  
True Effect ◽  
Meta Analyses

Abstract. Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates. Both of these problems lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect such bias in meta-analytic results. We present an evaluation of the performance of six of these tools. To assess the Type I error rate and the statistical power of these methods, we simulated a large variety of literatures that differed with regard to true effect size, heterogeneity, number of available primary studies, and sample sizes of these primary studies; furthermore, simulated studies were subjected to different degrees of publication bias. Our results show that across all simulated conditions, no method consistently outperformed the others. Additionally, all methods performed poorly when true effect sizes were heterogeneous or primary studies had a small chance of being published, irrespective of their results. This suggests that in many actual meta-analyses in psychology, bias will remain undiscovered no matter which detection method is used.

Which Data to Meta-Analyze, and How?

2019 ◽  
Vol 227 (1) ◽  
pp. 64-82 ◽  
Author(s):  
Martin Voracek ◽  
Michael Kossmeier ◽  
Ulrich S. Tran
Keyword(s):  
Empirical Research ◽  
Meta Analysis ◽  
Curve Analysis ◽  
Research Progress ◽  
Primary Data ◽  
Statistical Procedures ◽  
Graphical Displays ◽  
Primary Data Analysis ◽  
Meta Analyses ◽  
Data Analytic

Abstract. Which data to analyze, and how, are fundamental questions of all empirical research. As there are always numerous flexibilities in data-analytic decisions (a “garden of forking paths”), this poses perennial problems to all empirical research. Specification-curve analysis and multiverse analysis have recently been proposed as solutions to these issues. Building on the structural analogies between primary data analysis and meta-analysis, we transform and adapt these approaches to the meta-analytic level, in tandem with combinatorial meta-analysis. We explain the rationale of this idea, suggest descriptive and inferential statistical procedures, as well as graphical displays, provide code for meta-analytic practitioners to generate and use these, and present a fully worked real example from digit ratio (2D:4D) research, totaling 1,592 meta-analytic specifications. Specification-curve and multiverse meta-analysis holds promise to resolve conflicting meta-analyses, contested evidence, controversial empirical literatures, and polarized research, and to mitigate the associated detrimental effects of these phenomena on research progress.

Replication of the Superstition and Performance Study by

2014 ◽  
Vol 45 (3) ◽  
pp. 239-245 ◽  
Author(s):  
Robert J. Calin-Jageman ◽  
Tracy L. Caldwell
Keyword(s):  
Task Difficulty ◽  
Statistical Power ◽  
Meta Analysis ◽  
A Priori ◽  
Significant Heterogeneity ◽  
Performance Study ◽  
Improve Performance ◽  
Research Designs ◽  
Series Of Experiments ◽  
And Performance

A recent series of experiments suggests that fostering superstitions can substantially improve performance on a variety of motor and cognitive tasks ( Damisch, Stoberock, & Mussweiler, 2010 ). We conducted two high-powered and precise replications of one of these experiments, examining if telling participants they had a lucky golf ball could improve their performance on a 10-shot golf task relative to controls. We found that the effect of superstition on performance is elusive: Participants told they had a lucky ball performed almost identically to controls. Our failure to replicate the target study was not due to lack of impact, lack of statistical power, differences in task difficulty, nor differences in participant belief in luck. A meta-analysis indicates significant heterogeneity in the effect of superstition on performance. This could be due to an unknown moderator, but no effect was observed among the studies with the strongest research designs (e.g., high power, a priori sampling plan).

Power of Modified Brown-Forsythe and Mixed-Model Approaches in Split-Plot Designs

Methodology ◽  
2017 ◽  
Vol 13 (1) ◽  
pp. 9-22 ◽  
Author(s):  
Pablo Livacic-Rojas ◽  
Guillermo Vallejo ◽  
Paula Fernández ◽  
Ellián Tuero-Herrero
Keyword(s):  
Repeated Measures ◽  
Statistical Power ◽  
Mixed Model ◽  
Covariance Structure ◽  
Simulation Method ◽  
Future Research ◽  
Repeated Measures Design ◽  
Fixed And Random Effects ◽  
Split Plot ◽  
High Level

Abstract. Low precision of the inferences of data analyzed with univariate or multivariate models of the Analysis of Variance (ANOVA) in repeated-measures design is associated to the absence of normality distribution of data, nonspherical covariance structures and free variation of the variance and covariance, the lack of knowledge of the error structure underlying the data, and the wrong choice of covariance structure from different selectors. In this study, levels of statistical power presented the Modified Brown Forsythe (MBF) and two procedures with the Mixed-Model Approaches (the Akaike’s Criterion, the Correctly Identified Model [CIM]) are compared. The data were analyzed using Monte Carlo simulation method with the statistical package SAS 9.2, a split-plot design, and considering six manipulated variables. The results show that the procedures exhibit high statistical power levels for within and interactional effects, and moderate and low levels for the between-groups effects under the different conditions analyzed. For the latter, only the Modified Brown Forsythe shows high level of power mainly for groups with 30 cases and Unstructured (UN) and Autoregressive Heterogeneity (ARH) matrices. For this reason, we recommend using this procedure since it exhibits higher levels of power for all effects and does not require a matrix type that underlies the structure of the data. Future research needs to be done in order to compare the power with corrected selectors using single-level and multilevel designs for fixed and random effects.

The Limits of Ego Depletion

2019 ◽  
Vol 50 (5-6) ◽  
pp. 292-304 ◽  
Author(s):  
Mario Wenzel ◽  
Marina Lind ◽  
Zarah Rowland ◽  
Daniela Zahn ◽  
Thomas Kubiak
Keyword(s):  
Research Agenda ◽  
Statistical Power ◽  
Interindividual Variability ◽  
Ego Depletion ◽  
Crossover Design ◽  
Effect Sizes ◽  
Future Research ◽  
Future Research Agenda ◽  
Depletion Effect ◽  
Within Subjects

Abstract. Evidence on the existence of the ego depletion phenomena as well as the size of the effects and potential moderators and mediators are ambiguous. Building on a crossover design that enables superior statistical power within a single study, we investigated the robustness of the ego depletion effect between and within subjects and moderating and mediating influences of the ego depletion manipulation checks. Our results, based on a sample of 187 participants, demonstrated that (a) the between- and within-subject ego depletion effects only had negligible effect sizes and that there was (b) large interindividual variability that (c) could not be explained by differences in ego depletion manipulation checks. We discuss the implications of these results and outline a future research agenda.

Statistical Power in Meta-Analysis

2006 ◽  
Author(s):  
Guy Cafri ◽  
Michael T. Brannick ◽  
Jeffrey Kromrey
Keyword(s):  
Statistical Power ◽  
Meta Analysis
Sign in / Sign up

Export Citation Format

Share Document