scholarly journals Trainless model performance estimation based on random weights initialisations for neural architecture search

Array ◽  
2021 ◽  
pp. 100082
Author(s):  
Ekaterina Gracheva
Keyword(s):  
Model Performance ◽  
Performance Estimation ◽  
Neural Architecture ◽  
Random Weights

scholarly journals Confound Removal and Normalization in Practice: A Neuroimaging Based Sex Prediction Case Study

2021 ◽  
pp. 3-18
Author(s):  
Shammi More ◽  
Simon B. Eickhoff ◽  
Julian Caspers ◽  
Kaustubh R. Patil
Keyword(s):  
Brain Size ◽  
Model Performance ◽  
Performance Estimation ◽  
Systematic Evaluation ◽  
Biological Traits ◽  
Generalization Performance ◽  
Advantages And Disadvantages ◽  
Out Of Sample ◽  
Neuroimaging Data ◽  
Score Normalization

AbstractMachine learning (ML) methods are increasingly being used to predict pathologies and biological traits using neuroimaging data. Here controlling for confounds is essential to get unbiased estimates of generalization performance and to identify the features driving predictions. However, a systematic evaluation of the advantages and disadvantages of available alternatives is lacking. This makes it difficult to compare results across studies and to build deployment quality models. Here, we evaluated two commonly used confound removal schemes–whole data confound regression (WDCR) and cross-validated confound regression (CVCR)–to understand their effectiveness and biases induced in generalization performance estimation. Additionally, we study the interaction of the confound removal schemes with Z-score normalization, a common practice in ML modelling. We applied eight combinations of confound removal schemes and normalization (pipelines) to decode sex from resting-state functional MRI (rfMRI) data while controlling for two confounds, brain size and age. We show that both schemes effectively remove linear univariate and multivariate confounding effects resulting in reduced model performance with CVCR providing better generalization estimates, i.e., closer to out-of-sample performance than WDCR. We found no effect of normalizing before or after confound removal. In the presence of dataset and confound shift, four tested confound removal procedures yielded mixed results, raising new questions. We conclude that CVCR is a better method to control for confounding effects in neuroimaging studies. We believe that our in-depth analyses shed light on choices associated with confound removal and hope that it generates more interest in this problem instrumental to numerous applications.

scholarly journals Rethinking Performance Estimation in Neural Architecture Search

2020 ◽  
Author(s):  
Xiawu Zheng ◽  
Rongrong Ji ◽  
Qiang Wang ◽  
Qixiang Ye ◽  
Zhenguo Li ◽  
...  
Keyword(s):  
Performance Estimation ◽  
Neural Architecture

scholarly journals Neural Architecture Search for Lightweight Neural Network in Food Recognition

Mathematics ◽  
2021 ◽  
Vol 9 (11) ◽  
pp. 1245
Author(s):  
Ren Zhang Tan ◽  
XinYing Chew ◽  
Khai Wah Khaw
Keyword(s):  
Neural Network ◽  
Model Performance ◽  
Essential Element ◽  
Neural Model ◽  
Search Space ◽  
Recognition System ◽  
Experimental Result ◽  
Neural Architecture ◽  
Balanced Diet ◽  
Food Recognition

Healthy eating is an essential element to prevent obesity that will lead to chronic diseases. Despite numerous efforts to promote the awareness of healthy food consumption, the obesity rate has been increased in the past few years. An automated food recognition system is needed to serve as a fundamental source of information for promoting a balanced diet and assisting users to understand their meal consumption. In this paper, we propose a novel Lightweight Neural Architecture Search (LNAS) model to self-generate a thin Convolutional Neural Network (CNN) that can be executed on mobile devices with limited processing power. LNAS has a sophisticated search space and modern search strategy to design a child model with reinforcement learning. Extensive experiments have been conducted to evaluate the model generated by LNAS, namely LNAS-NET. The experimental result shows that the proposed LNAS-NET outperformed the state-of-the-art lightweight models in terms of training speed and accuracy metric. Those experiments indicate the effectiveness of LNAS without sacrificing the model performance. It provides a good direction to move toward the era of AutoML and mobile-friendly neural model design.

A New Virus Source Inference Model Based on Network Performance Estimation

2013 ◽  
Vol 433-435 ◽  
pp. 1693-1698
Author(s):  
Hai Ting Zhu ◽  
Wei Ding ◽  
Jun Hui Ni
Keyword(s):  
Network Performance ◽  
Performance Metrics ◽  
Model Performance ◽  
Performance Estimation ◽  
Estimation Accuracy ◽  
Inference Model ◽  
Model Based ◽  
Virus Source ◽  
Topology Information ◽  
Network Status

Combining network tomography, a new virus source inference model based on network performance is proposed. Both the topology information and real time network status are considered in our model. Performance metrics are introduced into rumor-centrality-based source detecting algorithm in an active inference way. By improving the process of setting up the spanning tree of infected topology we raise the virus source inference precision and reduce the time complexity of rumor-centrality-based algorithm from O(N2(|V|+|E|)) to O(N2). The simulation results show that our model achieves better estimation accuracy than the algorithm using rumor center as the estimator.

scholarly journals Accelerating multi-objective neural architecture search by random-weight evaluation

2021 ◽  
Author(s):  
Shengran Hu ◽  
Ran Cheng ◽  
Cheng He ◽  
Zhichao Lu ◽  
Jing Wang ◽  
...  
Keyword(s):  
Real World ◽  
High Performance ◽  
Performance Estimation ◽  
Stochastic Gradient Descent ◽  
Deep Convolutional Neural Networks ◽  
Efficient Manner ◽  
Multi Objective ◽  
Search Spaces ◽  
Neural Architecture ◽  
Random Weight

AbstractFor the goal of automated design of high-performance deep convolutional neural networks (CNNs), neural architecture search (NAS) methodology is becoming increasingly important for both academia and industries. Due to the costly stochastic gradient descent training of CNNs for performance evaluation, most existing NAS methods are computationally expensive for real-world deployments. To address this issue, we first introduce a new performance estimation metric, named random-weight evaluation (RWE) to quantify the quality of CNNs in a cost-efficient manner. Instead of fully training the entire CNN, the RWE only trains its last layer and leaves the remainders with randomly initialized weights, which results in a single network evaluation in seconds. Second, a complexity metric is adopted for multi-objective NAS to balance the model size and performance. Overall, our proposed method obtains a set of efficient models with state-of-the-art performance in two real-world search spaces. Then the results obtained on the CIFAR-10 dataset are transferred to the ImageNet dataset to validate the practicality of the proposed algorithm. Moreover, ablation studies on NAS-Bench-301 datasets reveal the effectiveness of the proposed RWE in estimating the performance compared to existing methods.

scholarly journals Uncertainty and risk in water quality modelling and management

2003 ◽  
Vol 5 (4) ◽  
pp. 259-274 ◽  
Author(s):  
Neil R. McIntyre ◽  
Thorsten Wagener ◽  
Howard S. Wheater ◽  
Zeng Si Yu
Keyword(s):  
Water Quality ◽  
Monte Carlo ◽  
Model Performance ◽  
Performance Estimation ◽  
Successful Implementation ◽  
Limited Information ◽  
Water Quality Modelling ◽  
Practical Applications ◽  
Modelling Framework ◽  
Successful Communication

The case is presented for increasing attention to the evaluation of uncertainty in water quality modelling practice, and for this evaluation to be extended to risk management applications. A framework for risk-based modelling of water quality is outlined and presented as a potentially valuable component of a broader risk assessment methodology. Technical considerations for the successful implementation of the modelling framework are discussed. The primary arguments presented are as follows. (1) For a large number of practical applications, deterministic use of complex water quality models is not supported by the available data and/or human resources, and is not warranted by the limited information contained in the results. Modelling tools should be flexible enough to be employed at levels of complexities which suit the modelling task, data and available resources. (2) Monte Carlo simulation has largely untapped potential for the evaluation of model performance, estimation of model uncertainty and identification of factors (including pollution sources, environmental influences and ill-defined objectives) contributing to the risk of failing water quality objectives. (3) For practical application of Monte Carlo methods, attention needs to be given to numerical efficiency, and for successful communication of results, effective interfaces are required. A risk-based modelling tool developed by the authors is introduced.

scholarly journals Transfer Learning by Similarity Centred Architecture Evolution for Multiple Residential Load Forecasting

Smart Cities ◽  
2021 ◽  
Vol 4 (1) ◽  
pp. 217-240
Author(s):  
Santiago Gomez-Rosero ◽  
Miriam A. M. Capretz ◽  
Syed Mir
Keyword(s):  
Evolutionary Algorithms ◽  
Transfer Learning ◽  
Low Voltage ◽  
Load Forecasting ◽  
Model Performance ◽  
Short Term ◽  
Neural Architecture ◽  
Wide Range ◽  
Short Term Load Forecasting ◽  
Architecture Evolution

The development from traditional low voltage grids to smart systems has become extensive and adopted worldwide. Expanding the demand response program to cover the residential sector raises a wide range of challenges. Short term load forecasting for residential consumers in a neighbourhood could lead to a better understanding of low voltage consumption behaviour. Nevertheless, users with similar characteristics can present diversity in consumption patterns. Consequently, transfer learning methods have become a useful tool to tackle differences among residential time series. This paper proposes a method combining evolutionary algorithms for neural architecture search with transfer learning to perform short term load forecasting in a neighbourhood with multiple household load consumption. The approach centres its efforts on neural architecture search using evolutionary algorithms. The neural architecture evolution process retains the patterns of the centre-most house, and later the architecture weights are adjusted for each house in a multihouse set from a neighbourhood. In addition, a sensitivity analysis was conducted to ensure model performance. Experimental results on a large dataset containing hourly load consumption for ten houses in London, Ontario showed that the performance of the proposed approach performs better than the compared techniques. Moreover, the proposed method presents the average accuracy performance of 3.17 points higher than the state-of-the-art LSTM one shot method.

Predictors of Purchasing a Hearing Aid After an Evaluation Period: A Prospective Study in Dutch Older Hearing Aid Candidates

2019 ◽  
Vol 28 (3S) ◽  
pp. 802-805 ◽  
Author(s):  
Marieke Pronk ◽  
Janine F. J. Meijerink ◽  
Sophia E. Kramer ◽  
Martijn W. Heymans ◽  
Jana Besser
Keyword(s):  
Help Seeking ◽  
Hearing Aid ◽  
Model Performance ◽  
Area Under The Curve ◽  
Secondary Data ◽  
Support Program ◽  
Evaluation Period ◽  
A Prospective Study ◽  
Study Participants ◽  
Baseline Covariates

Purpose The current study aimed to identify factors that distinguish between older (50+ years) hearing aid (HA) candidates who do and do not purchase HAs after having gone through an HA evaluation period (HAEP). Method Secondary data analysis of the SUpport PRogram trial was performed ( n = 267 older, 1st-time HA candidates). All SUpport PRogram participants started an HAEP shortly after study enrollment. Decision to purchase an HA by the end of the HAEP was the outcome of interest of the current study. Participants' baseline covariates (22 in total) were included as candidate predictors. Multivariable logistic regression modeling (backward selection and reclassification tables) was used. Results Of all candidate predictors, only pure-tone average (average of 1, 2, and 4 kHz) hearing loss emerged as a significant predictor (odds ratio = 1.03, 95% confidence interval [1.03, 1.17]). Model performance was weak (Nagelkerke R 2 = .04, area under the curve = 0.61). Conclusions These data suggest that, once HA candidates have decided to enter an HAEP, factors measured early in the help-seeking journey do not predict well who will and will not purchase an HA. Instead, factors that act during the HAEP may hold this predictive value. This should be examined.

Constructing and Deconstructing Concepts

2016 ◽  
Vol 63 (5) ◽  
pp. 249-262 ◽  
Author(s):  
Charles A. Doan ◽  
Ronaldo Vigo
Keyword(s):  
Information Theory ◽  
Model Performance ◽  
Unsupervised Classification ◽  
Family Resemblance ◽  
Category Membership ◽  
Formal Models ◽  
Human Beings ◽  
One Dimensional ◽  
Relational Patterns ◽  
Optimal Classification

Abstract. Several empirical investigations have explored whether observers prefer to sort sets of multidimensional stimuli into groups by employing one-dimensional or family-resemblance strategies. Although one-dimensional sorting strategies have been the prevalent finding for these unsupervised classification paradigms, several researchers have provided evidence that the choice of strategy may depend on the particular demands of the task. To account for this disparity, we propose that observers extract relational patterns from stimulus sets that facilitate the development of optimal classification strategies for relegating category membership. We conducted a novel constrained categorization experiment to empirically test this hypothesis by instructing participants to either add or remove objects from presented categorical stimuli. We employed generalized representational information theory (GRIT; Vigo, 2011b , 2013a , 2014 ) and its associated formal models to predict and explain how human beings chose to modify these categorical stimuli. Additionally, we compared model performance to predictions made by a leading prototypicality measure in the literature.

Sign in / Sign up

Export Citation Format

Share Document