scholarly journals Registered reports: an early example and analysis

PeerJ ◽  
2019 ◽  
Vol 7 ◽  
pp. e6232 ◽  
Author(s):  
Richard Wiseman ◽  
Caroline Watt ◽  
Diana Kornbrot
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Circumstantial Evidence ◽  
Focused Attention ◽  
Behavioral Sciences ◽  
Positive Role ◽  
Questionable Research Practices ◽  
Methodological Rigor ◽  
Replication Crisis ◽  
Crisis In Psychology

The recent ‘replication crisis’ in psychology has focused attention on ways of increasing methodological rigor within the behavioral sciences. Part of this work has involved promoting ‘Registered Reports’, wherein journals peer review papers prior to data collection and publication. Although this approach is usually seen as a relatively recent development, we note that a prototype of this publishing model was initiated in the mid-1970s by parapsychologist Martin Johnson in the European Journal of Parapsychology (EJP). A retrospective and observational comparison of Registered and non-Registered Reports published in the EJP during a seventeen-year period provides circumstantial evidence to suggest that the approach helped to reduce questionable research practices. This paper aims both to bring Johnson’s pioneering work to a wider audience, and to investigate the positive role that Registered Reports may play in helping to promote higher methodological and statistical standards.

Where Are The Self-Correcting Mechanisms In Science?

2020 ◽  
Author(s):  
Simine Vazire ◽  
Alex O. Holcombe
Keyword(s):  
Peer Review ◽  
Critical Appraisal ◽  
Scientific Field ◽  
The Self ◽  
Replication Crisis

It is often said that science is self-correcting, but the replication crisis suggests that, at least in some fields, self-correction mechanisms have fallen short of what we might hope for. How can we know whether a particular scientific field has effective self-correction mechanisms, that is, whether its findings are credible? The usual processes that supposedly provide mechanisms for scientific self-correction – mainly peer review and disciplinary committees – have been inadequate. We argue for more verifiable indicators of a field’s commitment to self-correction. These include transparency, which is already a target of many reform efforts, and critical appraisal, which has received less attention. Only by obtaining Measurements of Observable Self-Correction (MOSCs) can we begin to evaluate the claim that “science is self-correcting.” We expect the validity of this claim to vary across fields and subfields, and suggest that some fields, such as psychology and biomedicine, fall far short of an appropriate level of transparency and, especially, critical appraisal. Fields without robust, verifiable mechanisms for transparency and critical appraisal cannot reasonably be said to be self-correcting, and thus do not warrant the credibility often imputed to science as a whole.

Simplifying spectroscopic supplementary data collection

2017 ◽  
pp. 6-10
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Spectroscopic Data ◽  
Supplementary Data ◽  
Low Resolution ◽  
Full Data ◽  
The Future ◽  
Published Paper ◽  
Low Resolution Images

Tony Davies and a number of others consider collecting supplementary spectroscopic data. Like Eurospec, the plan is to use such supplementary data not only to enhance the published paper, but also to aid thorough peer-review by allowing reviewers access to the full data rather than, as Tony puts it, “low-resolution images of data”. I’m sure you will be interested in a look at the future through this column.

Psychology Exceptionalism and the Multiple Discovery of the Replication Crisis

2021 ◽  
pp. 108926802110465
Author(s):  
Nicole C. Nelson ◽  
Julie Chung ◽  
Kelsey Ichikawa ◽  
Momin M. Malik
Keyword(s):  
Subject Matter ◽  
Common Core ◽  
Early Years ◽  
Institutional Factors ◽  
Scientific Practices ◽  
Incentive Structure ◽  
Biomedical Sciences ◽  
Current Crisis ◽  
Replication Crisis ◽  
Crisis In Psychology

This article outlines what we call the “narrative of psychology exceptionalism” in commentaries on the replication crisis: many thoughtful commentaries link the current crisis to the specificity of psychology’s history, methods, and subject matter, but explorations of the similarities between psychology and other fields are comparatively thin. Historical analyses of the replication crisis in psychology further contribute to this exceptionalism by creating a genealogy of events and personalities that shares little in common with other fields. We aim to rebalance this narrative by examining the emergence and evolution of replication discussions in psychology alongside their emergence and evolution in biomedicine. Through a mixed-methods analysis of commentaries on replication in psychology and the biomedical sciences, we find that these conversations have, from the early years of the crisis, shared a common core that centers on concerns about the effectiveness of traditional peer review, the need for greater transparency in methods and data, and the perverse incentive structure of academia. Drawing on Robert Merton’s framework for analyzing multiple discovery in science, we argue that the nearly simultaneous emergence of this narrative across fields suggests that there are shared historical, cultural, or institutional factors driving disillusionment with established scientific practices.

scholarly journals Opportunities for increased reproducibility and replicability of developmental neuroimaging

2019 ◽  
Author(s):  
Eduard Klapwijk ◽  
Wouter van den Bos ◽  
Christian K. Tamnes ◽  
Nora Maria Raschle ◽  
Kathryn L. Mills
Keyword(s):  
Data Collection ◽  
Cognitive Neuroscience ◽  
Statistical Power ◽  
Large Scale ◽  
Small Samples ◽  
Developmental Studies ◽  
Data Analyses ◽  
Methodological Rigor ◽  
Research Findings ◽  
Tools And Techniques

Many workflows and tools that aim to increase the reproducibility and replicability of research findings have been suggested. In this review, we discuss the opportunities that these efforts offer for the field of developmental cognitive neuroscience, in particular developmental neuroimaging. We focus on issues broadly related to statistical power and to flexibility and transparency in data analyses. Critical considerations relating to statistical power include challenges in recruitment and testing of young populations, how to increase the value of studies with small samples, and the opportunities and challenges related to working with large-scale datasets. Developmental studies involve challenges such as choices about age groupings, lifespan modelling, analyses of longitudinal changes, and data that can be processed and analyzed in a multitude of ways. Flexibility in data acquisition, analyses and description may thereby greatly impact results. We discuss methods for improving transparency in developmental neuroimaging, and how preregistration can improve methodological rigor. While outlining challenges and issues that may arise before, during, and after data collection, solutions and resources are highlighted aiding to overcome some of these. Since the number of useful tools and techniques is ever-growing, we highlight the fact that many practices can be implemented stepwise.

scholarly journals Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

2020 ◽  
Vol 3 (3) ◽  
pp. 309-331 ◽  
Author(s):  
Charles R. Ebersole ◽  
Maya B. Mathur ◽  
Erica Baranski ◽  
Diane-Jo Bart-Plange ◽  
Nicholas R. Buttrick ◽  
...  
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Total Sample ◽  
Median Number ◽  
Open Science ◽  
Original Study ◽  
Effect Sizes ◽  
Replication Studies ◽  
Expert Review ◽  
Science Collaboration

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

Sign in / Sign up

Export Citation Format

Share Document