Publication Biases in Replication Studies

One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged over nonsignificant results. Recognition of this bias, along with increased calls for “open science,” has led to an emphasis on replication studies. Yet, few have explored publication bias and its consequences in replication studies. We offer a model of the publication process involving an initial study and a replication. We use the model to describe three types of publication biases: (1) file drawer bias, (2) a “repeat study” bias against the publication of replication studies, and (3) a “gotcha bias” where replication results that run contrary to a prior study are more likely to be published. We estimate the model’s parameters with a vignette experiment conducted with political science professors teaching at Ph.D. granting institutions in the United States. We find evidence of all three types of bias, although those explicitly involving replication studies are notably smaller. This bodes well for the replication movement. That said, the aggregation of all of the biases increases the number of false positives in a literature. We conclude by discussing a path for future work on publication biases.

Systematic Review and Meta-Analysis of Racial and Ethnic Differences in Dementia Caregivers’ Well-Being

Background and Objectives Studies comparing racial/ethnic differences on measures of psychological and physical well-being for dementia caregivers have reported differences between minority and white caregivers. Recruitment methods often differ for minority and white participants due to enrollment targets and may lead to biased comparisons, especially in convenience samples. We aimed to examine racial/ethnic differences in dementia caregiver outcomes and to determine whether differences vary between studies with population-based or convenience samples. Research Design and Methods We systematically reviewed articles with primary data from PubMed, Google Scholar, and PsycINFO. We included studies comparing African American or Hispanic/Latino to white dementia caregivers on measures of psychological well-being or physical well-being. Reviewers screened titles and abstracts, reviewed full texts and conducted risk-of-bias assessments. Meta-analyses were conducted to assess effects by race/ethnicity and study bias. Results A total of 159 effects were extracted from 38 studies, 2 of which were population based. Random-effects models revealed small but statistically significant effects with better psychological well-being in African American caregivers compared with white caregivers in both population-based (d = −0.22) and convenience sample studies (d = −0.21). Hispanics/Latino caregivers reported lower levels of physical well-being than white caregivers (d = 0.12), though these effects varied by level of rated study bias. Discussion and Implications Consistency across study methods raises confidence in the validity of previous reports of better psychological well-being in African American caregivers. Future studies should use population-based samples with subgroups of Hispanic/Latino, Asian American, and American Indian caregivers that are culturally distinct on factors such as country of origin and tribe.

Inference 1: Chance, bias, and confounding

Inference describes the process of deriving conclusions from observations to generalizations and is a key activity in all research. This chapter commences with considering how the findings from a research sample can be applied to the population from which that sample was drawn—that is, the role of chance in accounting for observed findings, and the possibility that they might have arisen because of errors in the design of the study (bias)—whether relating to the people in the sample (selection bias) or the measurements applied (information bias). The chapter then begins to consider the extent to which a causal relationship can be inferred from an observed association, by considering the role of confounding factors as alternative explanations and ways in which these are addressed in statistical analyses.

Assessing Confidence in the Results of Network Meta-Analysis (Cinema)

Evaluation of the credibility of results from a meta-analysis has become an intrinsic part of the evidence synthesis process. We present a methodological framework to evaluate Confidence In the results from Network Meta-Analysis (CINeMA) when multiple interventions are compared. CINeMA considers six domains and we outline the methods used to form judgements about within-study bias, across-studies bias, indirectness, imprecision, heterogeneity and incoherence. Key to judgements about within-study bias and indirectness is the percentage contribution matrix, which shows how much information each study contributes to the results from network meta-analysis. The use of contribution matrix allows the semi-automation of the process, implemented in a freely available web application (cinema.ispm.ch). In evaluating imprecision, heterogeneity and inconsistency we consider the impact of these components of variability in forming clinical decisions. Via three examples, we show that CINeMA improves transparency and avoids the selective use of evidence when forming judgements, thus limiting subjectivity in the process. CINeMA is easy to apply even in large and complicated networks, like a network involving 18 different antidepressant drugs.

Multidimensional internal dynamics underlying the perception of motion

When ambiguous visual stimuli are presented continuously, they often lead to oscillations between usually two perceptions. Because of these oscillations, it has been thought that the underlying neural dynamics also arises from a binary or two-state system. Contradicting the binary assumption, it has been shown recently that the perception of some ambiguous stimuli is governed by continuously varying internal states, measured as biases that differ considerably from one observer to the next and that can also evolve over time (Wexler et al., 2015). Here I study bias patterns in the motion quartet, an ambiguous apparent motion stimulus, as the quartet’s orientation is varied. The bias patterns are robustly idiosyncratic, and are even more complex than those that have been described previously. There are two qualitatively different bias types: some observers prefer a translation axis, while others show preference for a rotation direction. Each type also varies parametrically: the orientation of the preferred axis, and the direction of preferred rotation. The are also clear cases of combination of the two bias types. When measured repeatedly over 9 hours, the bias patterns usually remain stable, but also sometimes evolve both parametrically (e.g., change of preferred axis), as well as across bias type (change from axial to rotational bias). Control experiments revealed that the variety of bias patterns observed across subjects, and their changes over time, are not due to voluntary decisions. Overall, these results exhibit the multidimensional complexity of internal states underlying the perception of even simple stimuli.