Emerging Missing Data Estimation Problems

This chapter is divided into three parts: The first part presents a computational intelligence approach for predicting missing data in the presence of concept drift using an ensemble of multi-layered feed-forward neural networks. An algorithm that detects concept drift by measuring heteroskedasticity is proposed. Six instances prior to the occurrence of missing data are used to approximate the missing values. The algorithm is applied to simulated time series data sets resembling non-stationary data from a sensor. Results show that the prediction of missing data in non-stationary time series data is possible but is still a challenge. In the second part, an algorithm that uses dynamic programming and neural networks to solve the problem of missing data imputation is presented. A model that uses autoassociative neural networks and genetic algorithms is used as a basis; however, the neural networks are not trained using the entire data set. Data are broken up into granules and various models are created. The models are tested on a real dataset and the results show that the proposed method is effective in missing data estimation. In the third part of this chapter, a study of the impact of missing data estimation on fault classification in mechanical systems is undertaken. The fault classification task is implemented using the extension network as well as Gaussian mixture models. When the imputed values are used in the classification of faults using the extension networks, the fault classification accuracy of 95% is observed for single-missing-entry cases and 92% for two-missing-entry cases while the full database set is able to give classification accuracy of 97%. On the other hand, the Gaussian mixture model gives 94% for single-missing-entry cases and 92% for two-missing-entry cases while the full database set is able to give classification accuracy of 96%.

Note on a remarkable correlation between grain and straw, obtained at Rothamsted

A Unique correlation has recently been obtained from some experimental data at Rothamsted. In the autumn of 1924 a set of 47 unmanured plots were laid out in Sawyer's Field, and were sown with wheat to test the uniformity of the fertility of the soil. The plots were each ·098 acre. The produce of grain and straw from each was recorded, and an examination of the figures showed that the straw yield recorded for plot 1 A was much lower than one would have expected under normal circumstances. A careful enquiry into the matter revealed that by an oversight a weighing of straw had been removed from the scales before the weight had been entered in the record. As it was impossible to verify the exact value of the missing entry, the yields from this plot were omitted from all statistical reductions of the data, leaving 46 plots upon which to base an estimate of the various statistics.The soil was found to be very variable in fertility, the average yield of total grain per plot being 149·57 lb., with a standard error of 6·39 lb. and the average yield of total straw per plot was 194·35 lb., with a standard error of 7·55 lb.The grain and straw yields of the 46 plots are shown in the accompanying diagram. The very close association between the total weight of grain and total weight of straw is evident. The amount of the relationship was measured by correlating the 46 pairs of values.