Is your ad hoc model selection strategy affecting your multimodel inference?

Abstract An important but insufficiently addressed issue for machine learning in engineering applications is the task of model selection for new problems. Existing approaches to model selection generally focus on optimizing the learning algorithm and associated hyperparameters. However, in real-world engineering applications, the parameters that are external to the learning algorithm, such as feature engineering, can also have a significant impact on the performance of the model. These external parameters do not fit into most existing approaches for model selection and are therefore often studied ad hoc or not at all. In this article, we develop a statistical design of experiment (DOEs) approach to model selection based on the use of the Taguchi method. The key idea is that we use orthogonal arrays to plan a set of build-and-test experiments to study the external parameters in combination with the learning algorithm. The use of orthogonal arrays maximizes the information learned from each experiment and, therefore, enables the experimental space to be explored extremely efficiently in comparison with grid or random search methods. We demonstrated the application of the statistical DOE approach to a real-world model selection problem involving predicting service request escalation. Statistical DOE significantly reduced the number of experiments necessary to fully explore the external parameters for this problem and was able to successfully optimize the model with respect to the objective function of minimizing total cost in addition to the standard evaluation metrics such as accuracy, f-measure, and g-mean.

Download Full-text

A multimodel inference approach to categorical variant choice: construction, priming and frequency effects on the choice between full and contracted forms of am, are and is

Corpus Linguistics and Linguistic Theory ◽

10.1515/cllt-2014-0022 ◽

2017 ◽

Vol 13 (2) ◽

pp. 203-260 ◽

Cited By ~ 12

Author(s):

Danielle Barth ◽

Vsevolod Kapatsinski

Keyword(s):

Model Selection ◽

Prior Work ◽

Single Model ◽

Frequency Effects ◽

Multimodel Inference ◽

Classic Problem ◽

Model Selection Uncertainty ◽

Corpus Data ◽

Selection For ◽

Auxiliary Contraction

AbstractThe present paper presents a multimodel inference approach to linguistic variation, expanding on prior work by Kuperman and Bresnan (2012). We argue that corpus data often present the analyst with high model selection uncertainty. This uncertainty is inevitable given that language is highly redundant: every feature is predictable from multiple other features. However, uncertainty involved in model selection is ignored by the standard method of selecting the single best model and inferring the effects of the predictors under the assumption that the best model is true. Multimodel inference avoids committing to a single model. Rather, we make predictions based on the entire set of plausible models, with contributions of models weighted by the models' predictive value. We argue that multimodel inference is superior to model selection for both the I-Language goal of inferring the mental grammars that generated the corpus, and the E-Language goal of predicting characteristics of future speech samples from the community represented by the corpus. Applying multimodel inference to the classic problem of English auxiliary contraction, we show that the choice between multimodel inference and model selection matters in practice: the best model may contain predictors that are not significant when the full set of plausible models is considered, and may omit predictors that are significant considering the full set of models. We also contribute to the study of English auxiliary contraction. We document the effects of priming, contextual predictability, and specific syntactic constructions and provide evidence against effects of phonological context.

Download Full-text

A novel machine learning-based electricity price forecasting model based on optimal model selection strategy

Energy ◽

10.1016/j.energy.2021.121989 ◽

2022 ◽

Vol 238 ◽

pp. 121989

Author(s):

Wendong Yang ◽

Shaolong Sun ◽

Yan Hao ◽

Shouyang Wang

Keyword(s):

Machine Learning ◽

Model Selection ◽

Electricity Price ◽

Selection Strategy ◽

Forecasting Model ◽

Price Forecasting ◽

Optimal Model ◽

Model Based ◽

Electricity Price Forecasting

Download Full-text