Sample size planning for survival prediction with focus on high-dimensional data

Constructions of data driven ordering of set of multivariate observations are presented. The methods employ also dissimilarity measures. The ranks are used in the construction of test statistics for location problem and in the construction of the corresponding multiple comparisons rule. An important aspect of the resulting procedures is that they can be used also in the multisample setting and in situations where the sample size is smaller than the dimension of the observations. The performance of the proposed procedures is illustrated by simulations.

Download Full-text

A Two-Sample Test for Mean Vectors in High-Dimensional Data

Applied Science and Innovative Research ◽

10.22158/asir.v1n2p118 ◽

2017 ◽

Vol 1 (2) ◽

pp. 118

Author(s):

Knavoot Jiamwattanapong ◽

Samruam Chongcharoen

Keyword(s):

Sample Size ◽

High Dimensional Data ◽

Null Distribution ◽

High Dimensional ◽

Measurement Technology ◽

Test Statistic ◽

Dna Microarray Data ◽

Statistical Inferences ◽

Sample Test ◽

Mean Vectors

Modern measurement technology has enabled the capture of high-dimensional data by researchers and statisticians and classical statistical inferences, such as the renowned Hotelling’s T2 test, are no longer valid when the dimension of the data equals or exceeds the sample size. Importantly, when correlations among variables in a dataset exist, taking them into account in the analysis method would provide more accurate conclusions. In this article, we consider the hypothesis testing problem for two mean vectors in high-dimensional data with an underlying normality assumption. A new test is proposed based on the idea of keeping more information from the sample covariances. The asymptotic null distribution of the test statistic is derived. The simulation results show that the proposed test performs well comparing with other competing tests and becomes more powerful when the dimension increases for a given sample size. The proposed test is also illustrated with an analysis of DNA microarray data.

Download Full-text

Supervised t-Distributed Stochastic Neighbor Embedding for Data Visualization and Classification

INFORMS Journal on Computing ◽

10.1287/ijoc.2020.0961 ◽

2020 ◽

Author(s):

Yichen Cheng ◽

Xinlei Wang ◽

Yusen Xia

Keyword(s):

Dimension Reduction ◽

Sample Size ◽

High Dimensional Data ◽

Machine Learning Algorithms ◽

High Dimensional ◽

Data Sets ◽

Wearable Electronics ◽

Processing Technologies ◽

Leibler Divergence ◽

Supervised Dimension Reduction

We propose a novel supervised dimension-reduction method called supervised t-distributed stochastic neighbor embedding (St-SNE) that achieves dimension reduction by preserving the similarities of data points in both feature and outcome spaces. The proposed method can be used for both prediction and visualization tasks with the ability to handle high-dimensional data. We show through a variety of data sets that when compared with a comprehensive list of existing methods, St-SNE has superior prediction performance in the ultrahigh-dimensional setting in which the number of features p exceeds the sample size n and has competitive performance in the p ≤ n setting. We also show that St-SNE is a competitive visualization tool that is capable of capturing within-cluster variations. In addition, we propose a penalized Kullback–Leibler divergence criterion to automatically select the reduced-dimension size k for St-SNE. Summary of Contribution: With the fast development of data collection and data processing technologies, high-dimensional data have now become ubiquitous. Examples of such data include those collected from environmental sensors, personal mobile devices, and wearable electronics. High-dimensionality poses great challenges for data analytics routines, both methodologically and computationally. Many machine learning algorithms may fail to work for ultrahigh-dimensional data, where the number of the features p is (much) larger than the sample size n. We propose a novel method for dimension reduction that can (i) aid the understanding of high-dimensional data through visualization and (ii) create a small set of good predictors, which is especially useful for prediction using ultrahigh-dimensional data.

Download Full-text

-Plot for Testing Spherical Symmetry for High-Dimensional Data with a Small Sample Size

Journal of Probability and Statistics ◽

10.1155/2012/728565 ◽

2012 ◽

Vol 2012 ◽

pp. 1-18

Author(s):

Jiajuan Liang

Keyword(s):

Sample Size ◽

Spherical Symmetry ◽

Graphical Method ◽

Small Sample Size ◽

High Dimensional Data ◽

Monte Carlo Study ◽

Real Data ◽

Small Sample ◽

High Dimensional ◽

Data Set

High-dimensional data with a small sample size, such as microarray data and image data, are commonly encountered in some practical problems for which many variables have to be measured but it is too costly or time consuming to repeat the measurements for many times. Analysis of this kind of data poses a great challenge for statisticians. In this paper, we develop a new graphical method for testing spherical symmetry that is especially suitable for high-dimensional data with small sample size. The new graphical method associated with the local acceptance regions can provide a quick visual perception on the assumption of spherical symmetry. The performance of the new graphical method is demonstrated by a Monte Carlo study and illustrated by a real data set.

Download Full-text

High-Dimensional Statistical Learning: Roots, Justifications, and Potential Machineries

Cancer Informatics ◽

10.4137/cin.s30804 ◽

2015 ◽

Vol 14s5 ◽

pp. CIN.S30804 ◽

Cited By ~ 2

Author(s):

Amin Zollanvari

Keyword(s):

Sample Size ◽

Statistical Learning ◽

High Dimensional Data ◽

Operating Conditions ◽

High Dimensional ◽

Finite Sample ◽

Statistical Software ◽

Software Packages ◽

State Of Affairs ◽

Remarkable Progress

High-dimensional data generally refer to data in which the number of variables is larger than the sample size. Analyzing such datasets poses great challenges for classical statistical learning because the finite-sample performance of methods developed within classical statistical learning does not live up to classical asymptotic premises in which the sample size unboundedly grows for a fixed dimensionality of observations. Much work has been done in developing mathematical-statistical techniques for analyzing high-dimensional data. Despite remarkable progress in this field, many practitioners still utilize classical methods for analyzing such datasets. This state of affairs can be attributed, in part, to a lack of knowledge and, in part, to the ready-to-use computational and statistical software packages that are well developed for classical techniques. Moreover, many scientists working in a specific field of high-dimensional statistical learning are either not aware of other existing machineries in the field or are not willing to try them out. The primary goal in this work is to bring together various machineries of high-dimensional analysis, give an overview of the important results, and present the operating conditions upon which they are grounded. When appropriate, readers are referred to relevant review articles for more information on a specific subject.

Download Full-text