Bio-equivalence tests in functional data by maximum deviation

We study the problem of testing the equivalence of functional parameters, such as the mean or variance function, in the two sample functional data problem. In contrast to previous work, which reduces the functional problem to a multiple testing problem for the equivalence of scalar data by comparing the functions at each point, our approach is based on an estimate of a distance measuring the maximum deviation between the two functional parameters. Equivalence is claimed if the estimate for the maximum deviation does not exceed a given threshold. A bootstrap procedure is proposed to obtain quantiles for the distribution of the test statistic and consistency of the corresponding test is proved in the large sample scenario. As the methods proposed here avoid the use of the intersection-union principle they are less conservative and more powerful than the currently available methodology.

An experimentally-derived measure of inter-replicate variation in reference samples: the same-same permutation methodology

The multiple testing problem is a well-known statistical stumbling block in high-throughput data analysis, where large scale repetition of statistical methods introduces unwanted noise into the results. While approaches exist to overcome the multiple testing problem, these methods focus on theoretical statistical clarification rather than incorporating experimentally-derived measures to ensure appropriately tailored analysis parameters. Here, we introduce a method for estimating inter-replicate variability in reference samples for a quantitative proteomics experiment using permutation analysis. This can function as a modulator to multiple testing corrections such as the Benjamini-Hochberg ordered Q value test. We refer to this as a ‘same-same’ analysis, since this method incorporates the use of six biological replicates of the reference sample and determines, through non-redundant triplet pairwise comparisons, the level of quantitative noise inherent within the system. The method can be used to produce an experiment-specific Q value cut-off that achieves a specified false discovery rate at the quantitation level, such as 1%. The same-same method is applicable to any experimental set that incorporates six replicates of a reference sample. To facilitate access to this approach, we have developed a same-same analysis R module that is freely available and ready to use via the internet.

Multiple testing correction over contrasts for brain imaging

The multiple testing problem arises not only when there are many voxels or vertices in an image representation of the brain, but also when multiple contrasts of parameter estimates (that is, hypotheses) are tested in the same general linear model. Here we argue that a correction for this multiplicity must be performed to avoid excess of false positives. Various methods have been proposed in the literature, but few have been applied to brain imaging. Here we discuss and compare different methods to make such correction in different scenarios, showing that one classical and well known method is invalid, and argue that permutation is the best option to perform such correction due to its exactness and flexibility to handle a variety of common imaging situations.

An Algorithm for Early Outbreak Detection in Multiple Data Streams

ObjectiveTo propose a computationally simple, fast, and reliable temporal method for early event detection in multiple data streamsIntroductionCurrent biosurveillance systems run multiple univariate statistical process control (SPC) charts to detect increases in multiple data streams1. The method of using multiple univariate SPC charts is easy to implement and easy to interpret. By examining alarms from each control chart, it is easy to identify which data stream is causing the alarm. However, testing multiple data streams simultaneously can lead to multiple testing problems that inflate the combined false alarm probability. Although methods such as the Bonferroni correction can be applied to address the multiple testing problem by lowering the false alarm probability in each control chart, these approaches can be extremely conservative.Biosurveillance systems often make use of variations of popular univariate SPC charts such as the Shewart Chart, the cumulative sum chart (CUSUM), and the exponentially weighted moving average chart (EWMA). In these control charts an alarm is signaled when the charting statistic exceeds a pre-defined control limit. With the standard SPC charts, the false alarm rate is specified using the in-control average run length (ARL0). If multiple charts are used, the resulting multiple testing problem is often addressed using family-wise error rate (FWER) based methods – that are known to be conservative - for error control.A new temporal method is proposed for early event detection in multiple data streams. The proposed method uses p-values instead of the control limits that are commonly used with standard SPC charts. In addition, the proposed method uses false discovery rate (FDR) for error control over the standard ARL0 used with conventional SPC charts. With the use of FDR for error control, the proposed method makes use of more powerful and up-to-date procedures for handling the multiple testing problem than FWER-based methods.MethodsThe proposed method can be applied to multiple univariate CUSUM or EWMA control charts. It can also be applied to a variation of the Hotelling T2 chart which is a common multivariate process monitoring method. The Hotelling T2 chart is analogous to the Shewart chart. Montgomery et. al2 proposed a variation of the Hotelling T2 chart where the T2 statistic is decomposed into components that reflect the contribution of each data stream.First, a tolerable FDR level specified. Then, at each new time step disease counts from each of the m geographic regions Y1t, Y2t, … , Ymt are collected. These disease counts are used to calculate the charting statistics S1t, S2t, … , Smt for each region. Meanwhile by inspecting historical data from each region, a non-outbreak period is identified. Using data from the non-outbreak period, bootstrap samples are drawn with replacement from each region and charting statistics are calculated. Using the charting statistics, empirical non-outbreak distributions are generated for each region. With the empirical non-outbreak distributions and the current charting statistic for each region S1t, S2t, … , Smt , corresponding p-values p1t, p2t, … , pmt are calculated. The multiple testing problem that occurs in comparing multiple p-values simultaneously is handled using the Storey -Tibshirani multiple comparison procedure3 to signal alarms.ResultsAs an illustration, all three methods – EWMA, CUSUM, and Hotelling T2 (components) - were applied to a data set consisting of weekly disease count data from 16 German federal sates gathered over a 11 year period from 2004-2014. The first two years of data from 2004-2005 were used to calibrate the model. Figure 1 shows the results for the state of Rhineland Palatinate. The three plots in Figure 1 show (a) the weekly disease counts for Rhineland Palatinate (b) the EWMA statistic (shown in red), the CUSUM statistic (shown in dark green) and (c) the component of the Hotelling T2 statistic corresponding to the illustrated state (shown in blue). The actual outbreak occurred on week 306 (shown by the orange line). Notice the two false alarms – alarms that occur before week 306 - with the Hotelling T2 statistic (dark green) on weeks 34 and 292; similarly, the CUSUM statistic signals a false alarm on week 57. However, the EWMA statistic does not signal any false alarms before the outbreak (red). Figure 2 zooms on the alarm statistics for the time period from weeks 280 – 330. The Hotelling T2 statistic misses the onset of actual outbreak on week 306. The CUSUM statistic detects the outbreak on week 307 – one week later. However, the EWMA statistic detects the outbreak right at the onset on week 306.ConclusionsExtensive simulation studies were conducted to compare the performance of the three control charts. Performance was compared in terms of (i) speed of detection and (ii) false alarm rates. Simulation results provide convincing evidence that the EWMA and the CUSUM are considerably speedier in detecting outbreaks compared to Hotelling T2 statistic: compared to the CUSUM, the EWMA is relatively faster. Similarly, the false alarm rates are larger for Hotelling T2 statistic compared to the EWMA and the CUSUM: false alarms are rare with both the EWMA and the CUSUM statistics with EWMA statistic having a slight edge. Overall, EWMA has the best performance out of the three methods with the new algorithm. Thus, the new algorithm applied to the EWMA statistic provides a simple, fast, and a reliable method for early event detection in multiple data streams.References1. Fricker RD. Introduction to Statistical Methods for Biosurveillance. New York, NY: Cambridge University Press; 2013. 399p.2. Runger GC, Alt FB, Montgomery DC. Contributors to Multivariate Statistical Process Control Signal. Communications in Statistics – Theory and Methods. 1996; 25(10): 2203-2213.3. Storey JD, Tibshirani R. Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences USA 2003; 100:9440–9445.

Using Shrinkage in Multilevel Models to Understand Intersectionality

Multilevel models have recently been used to empirically investigate the idea that social characteristics are intersectional such as age, sex, ethnicity, and socioeconomic position interact with each other to drive outcomes. Some argue this approach solves the multiple-testing problem found in standard dummy-variable (fixed-effects) regression, because intersectional effects are automatically shrunk toward their mean. The hope is intersections appearing statistically significant by chance in a fixed-effects regression will not appear so in a multilevel model. However, this requires assumptions that are likely to be broken. We use simulations to show the effect of breaking these assumptions: when there are true main effects/interactions, unmodeled in the fixed part of the model. We show, while the multilevel approach outperforms the fixed-effects approach, shrinkage is less than is desired, and some intersectional effects are likely to appear erroneously statistically significant by chance. We conclude with advice to make this promising method work robustly.

Personality Reflection in the Brain’s Intrinsic Functional Architecture Remains Elusive

In the last years, there has been a considerable increase of research into the neuroimaging correlates of inter-individual temperament and character variability - an endeavour for which the term ‘personality neuroscience’ was coined. Among other neuroimaging modalities and approaches, substantial work focuses on functional connectivity in resting state (rs-FC) functional magnetic resonance imaging data. In the current paper, we set out to replicate a highly cited study that reported a range of functional connectivity correlates of personality dimensions assessed by the widely used ‘Big Five’ Personality Inventory. Using a larger sample (84 subjects) and an equivalent data analysis pipeline, we obtained widely disagreeing results compared to the original study. Overall, the results were in line with the hypotheses of no relation between functional connectivity and personality, when more precise permutation-based multiple testing procedures were applied. The results demonstrate that as with other neuroimaging studies, great caution should be applied when interpreting the findings, among other reasons due to multiple testing problem involved at several levels in many neuroimaging studies. Of course, the current study results can not ultimately disprove the existence of some link between personality and brain’s intrinsic functional architecture, but clearly shows that its form is very likely different and much more subtle and elusive than was previously reported.

Estimating the Proportion of True Null Hypotheses in Multiple Testing Problems

The problem of estimating the proportion, π0, of the true null hypotheses in a multiple testing problem is important in cases where large scale parallel hypotheses tests are performed independently. While the problem is a quantity of interest in its own right in applications, the estimate of π0 can be used for assessing or controlling an overall false discovery rate. In this article, we develop an innovative nonparametric maximum likelihood approach to estimate π0. The nonparametric likelihood is proposed to be restricted to multinomial models and an EM algorithm is also developed to approximate the estimate of π0. Simulation studies show that the proposed method outperforms other existing methods. Using experimental microarray datasets, we demonstrate that the new method provides satisfactory estimate in practice.