Multivariate Time Series Data Prediction Based on ATT-LSTM Network

Jie Ju; Fang-Ai Liu

doi:10.3390/app11209373

Multivariate Time Series Data Prediction Based on ATT-LSTM Network

Applied Sciences ◽

10.3390/app11209373 ◽

2021 ◽

Vol 11 (20) ◽

pp. 9373

Author(s):

Jie Ju ◽

Fang-Ai Liu

Keyword(s):

Time Series ◽

Deep Learning ◽

Time Series Data ◽

Mutual Influence ◽

Multivariate Time Series ◽

The Other ◽

Series Data ◽

Data Sets ◽

Learning Models ◽

Prediction Problems

Deep learning models have been widely used in prediction problems in various scenarios and have shown excellent prediction effects. As a deep learning model, the long short-term memory neural network (LSTM) is potent in predicting time series data. However, with the advancement of technology, data collection has become more accessible, and multivariate time series data have emerged. Multivariate time series data are often characterized by a large amount of data, tight timeline, and many related sequences. Especially in real data sets, the change rules of many sequences will be affected by the changes of other sequences. The interacting factors data, mutation information, and other issues seriously impact the prediction accuracy of deep learning models when predicting this type of data. On the other hand, we can also extract the mutual influence information between different sequences and simultaneously use the extracted information as part of the model input to make the prediction results more accurate. Therefore, we propose an ATT-LSTM model. The network applies the attention mechanism (attention) to the LSTM to filter the mutual influence information in the data when predicting the multivariate time series data, which makes up for the poor ability of the network to process data. Weaknesses have greatly improved the accuracy of the network in predicting multivariate time series data. To evaluate the model’s accuracy, we compare the ATT-LSTM model with the other six models on two real multivariate time series data sets based on two evaluation indicators: Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). The experimental results show that the model has an excellent performance improvement compared with the other six models, proving the model’s effectiveness in predicting multivariate time series data.

Download Full-text

Generation of Time-Series Working Patterns for Manufacturing High-Quality Products through Auxiliary Classifier Generative Adversarial Network

Sensors ◽

10.3390/s22010029 ◽

2021 ◽

Vol 22 (1) ◽

pp. 29

Author(s):

Manas Bazarbaev ◽

Tserenpurev Chuluunsaikhan ◽

Hyoseok Oh ◽

Ga-Ae Ryu ◽

Aziz Nasridinov ◽

...

Keyword(s):

Time Series ◽

Deep Learning ◽

Product Quality ◽

Time Series Data ◽

The Other ◽

Series Data ◽

Learning Models ◽

Generative Adversarial Network ◽

Adversarial Network ◽

Proposed Model

Product quality is a major concern in manufacturing. In the metal processing industry, low-quality products must be remanufactured, which requires additional labor, money, and time. Therefore, user-controllable variables for machines and raw material compositions are key factors for ensuring product quality. In this study, we propose a method for generating the time-series working patterns of the control variables for metal-melting induction furnaces and continuous casting machines, thus improving product quality by aiding machine operators. We used an auxiliary classifier generative adversarial network (AC-GAN) model to generate time-series working patterns of two processes depending on product type and additional material data. To check accuracy, the difference between the generated time-series data of the model and the ground truth data was calculated. Specifically, the proposed model results were compared with those of other deep learning models: multilayer perceptron (MLP), convolutional neural network (CNN), long short-term memory (LSTM), and gated recurrent unit (GRU). It was demonstrated that the proposed model outperformed the other deep learning models. Moreover, the proposed method generated different time-series data for different inputs, whereas the other deep learning models generated the same time-series data.

Download Full-text

Implementation of IoT Framework with Data Analysis Using Deep Learning Methods for Occupancy Prediction in a Building

Future Internet ◽

10.3390/fi13030067 ◽

2021 ◽

Vol 13 (3) ◽

pp. 67

Author(s):

Eric Hitimana ◽

Gaurav Bajpai ◽

Richard Musabe ◽

Louis Sibomana ◽

Jayavel Kayalvizhi

Keyword(s):

Machine Learning ◽

Time Series ◽

Deep Learning ◽

Time Series Data ◽

Multivariate Time Series ◽

Machine Learning Algorithms ◽

Series Data ◽

Support Vector ◽

Human Beings ◽

Feed Forward Network

Many countries worldwide face challenges in controlling building incidence prevention measures for fire disasters. The most critical issues are the localization, identification, detection of the room occupant. Internet of Things (IoT) along with machine learning proved the increase of the smartness of the building by providing real-time data acquisition using sensors and actuators for prediction mechanisms. This paper proposes the implementation of an IoT framework to capture indoor environmental parameters for occupancy multivariate time-series data. The application of the Long Short Term Memory (LSTM) Deep Learning algorithm is used to infer the knowledge of the presence of human beings. An experiment is conducted in an office room using multivariate time-series as predictors in the regression forecasting problem. The results obtained demonstrate that with the developed system it is possible to obtain, process, and store environmental information. The information collected was applied to the LSTM algorithm and compared with other machine learning algorithms. The compared algorithms are Support Vector Machine, Naïve Bayes Network, and Multilayer Perceptron Feed-Forward Network. The outcomes based on the parametric calibrations demonstrate that LSTM performs better in the context of the proposed application.

Download Full-text

Intra-domain and cross-domain transfer learning for time series

10.5194/egusphere-egu21-12142 ◽

2021 ◽

Author(s):

Erik Otović ◽

Marko Njirjak ◽

Dario Jozinović ◽

Goran Mauša ◽

Alberto Michelini ◽

...

Keyword(s):

Machine Learning ◽

Time Series ◽

Transfer Learning ◽

Time Series Data ◽

The Other ◽

Series Data ◽

Sound Recognition ◽

Transfer Of Knowledge ◽

Learning Models ◽

Machine Learning Models

In this study, we compared the performance of machine learning models trained using transfer learning and those that were trained from scratch - on time series data. Four machine learning models were used for the experiment. Two models were taken from the field of seismology, and the other two are general-purpose models for working with time series data. The accuracy of selected models was systematically observed and analyzed when switching within the same domain of application (seismology), as well as between mutually different domains of application (seismology, speech, medicine, finance). In seismology, we used two databases of local earthquakes (one in counts, and the other with the instrument response removed) and a database of global earthquakes for predicting earthquake magnitude; other datasets targeted classifying spoken words (speech), predicting stock prices (finance) and classifying muscle movement from EMG signals (medicine). In practice, it is very demanding and sometimes impossible to collect datasets of tagged data large enough to successfully train a machine learning model. Therefore, in our experiment, we use reduced data sets of 1,500 and 9,000 data instances to mimic such conditions. Using the same scaled-down datasets, we trained two sets of machine learning models: those that used transfer learning for training and those that were trained from scratch. We compared the performances between pairs of models in order to draw conclusions about the utility of transfer learning. In order to confirm the validity of the obtained results, we repeated the experiments several times and applied statistical tests to confirm the significance of the results. The study shows when, within the set experimental framework, the transfer of knowledge brought improvements in terms of model accuracy and in terms of model convergence rate. Our results show that it is possible to achieve better performance and faster convergence by transferring knowledge from the domain of global earthquakes to the domain of local earthquakes; sometimes also vice versa. However, improvements in seismology can sometimes also be achieved by transferring knowledge from medical and audio domains. The results show that the transfer of knowledge between other domains brought even more significant improvements, compared to those within the field of seismology. For example, it has been shown that models in the field of sound recognition have achieved much better performance compared to classical models and that the domain of sound recognition is very compatible with knowledge from other domains. We came to similar conclusions for the domains of medicine and finance. Ultimately, the paper offers suggestions when transfer learning is useful, and the explanations offered can provide a good starting point for knowledge transfer using time series data.

Download Full-text

Corporation financial distress prediction with deep learning: analysis of public listed companies in Malaysia

Business Process Management Journal ◽

10.1108/bpmj-06-2020-0273 ◽

2021 ◽

Vol ahead-of-print (ahead-of-print) ◽

Author(s):

Zulkifli Halim ◽

Shuhaida Mohamed Shuhidan ◽

Zuraidah Mohd Sanusi

Keyword(s):

Time Series ◽

Deep Learning ◽

Financial Distress ◽

Time Series Data ◽

Series Data ◽

Learning Models ◽

Content Type ◽

Financial Distress Prediction ◽

Distress Prediction ◽

Gated Recurrent Unit

PurposeIn the previous study of financial distress prediction, deep learning techniques performed better than traditional techniques over time-series data. This study investigates the performance of deep learning models: recurrent neural network, long short-term memory and gated recurrent unit for the financial distress prediction among the Malaysian public listed corporation over the time-series data. This study also compares the performance of logistic regression, support vector machine, neural network, decision tree and the deep learning models on single-year data.Design/methodology/approachThe data used are the financial data of public listed companies that been classified as PN17 status (distress) and non-PN17 (not distress) in Malaysia. This study was conducted using machine learning library of Python programming language.FindingsThe findings indicate that all deep learning models used for this study achieved 90% accuracy and above with long short-term memory (LSTM) and gated recurrent unit (GRU) getting 93% accuracy. In addition, deep learning models consistently have good performance compared to the other models over single-year data. The results show LSTM and GRU getting 90% and recurrent neural network (RNN) 88% accuracy. The results also show that LSTM and GRU get better precision and recall compared to RNN. The findings of this study show that the deep learning approach will lead to better performance in financial distress prediction studies. To be added, time-series data should be highlighted in any financial distress prediction studies since it has a big impact on credit risk assessment.Research limitations/implicationsThe first limitation of this study is the hyperparameter tuning only applied for deep learning models. Secondly, the time-series data are only used for deep learning models since the other models optimally fit on single-year data.Practical implicationsThis study proposes recommendations that deep learning is a new approach that will lead to better performance in financial distress prediction studies. Besides that, time-series data should be highlighted in any financial distress prediction studies since the data have a big impact on the assessment of credit risk.Originality/valueTo the best of authors' knowledge, this article is the first study that uses the gated recurrent unit in financial distress prediction studies based on time-series data for Malaysian public listed companies. The findings of this study can help financial institutions/investors to find a better and accurate approach for credit risk assessment.

Download Full-text

Assessment of Deep Learning Models for Human Activity Recognition on Multi-variate Time Series Data and Non-targeted Adversarial Attack

10.1007/978-3-030-78124-8_6 ◽

2021 ◽

pp. 129-159

Author(s):

Mahbuba Tasmin ◽

Sharif Uddin Ruman ◽

Taoseef Ishtiak ◽

Arif-ur-Rahman Chowdhury Suhan ◽

Redwan Hasif ◽

...

Keyword(s):

Time Series ◽

Deep Learning ◽

Activity Recognition ◽

Human Activity ◽

Time Series Data ◽

Human Activity Recognition ◽

Series Data ◽

Learning Models ◽

Adversarial Attack

Download Full-text

Remaining Useful Life Estimation and Prognostication of SAC305 PCB's for Dynamic Conditions of Temperature and Vibration Loads

Journal of Electronic Packaging ◽

10.1115/1.4052630 ◽

2021 ◽

Author(s):

Pradeep Lall ◽

Tony Thomas ◽

Ken Blecker

Keyword(s):

Time Series ◽

Deep Learning ◽

Time Series Data ◽

Short Term Memory ◽

Multivariate Time Series ◽

Remaining Useful Life ◽

Series Data ◽

Sac305 Solder ◽

Feature Vectors ◽

Useful Life

Abstract This study focuses on the feature vector identification and Remaining Useful Life (RUL) estimation of SAC305 solder alloy PCB's of two different configurations during varying conditions of temperature and vibration. The feature vectors are identified using the strain signals acquired from four symmetrical locations of the PCB at regular intervals during vibration. Two different types of experiments are employed to characterize the PCB's dynamic changes with varying temperature and acceleration levels. The strain signals acquired during each of these experiments are compared based on both time and frequency domain characteristics. Different statistical and frequency-based techniques were used to identify the strain signal variations with changes in the environment and loading conditions. The feature vectors in predicting failure at a constant working temperature and load were identified, and as an extension to this work, the effectiveness of the feature vectors during varying conditions of temperature and acceleration levels are investigated. The remaining Useful Life of the packages was estimated using a deep learning approach based on Long Short Term Memory (LSTM) network. This technique can identify the underlying patterns in multivariate time series data that can predict the packages' life. The autocorrelation function's residuals were used as the multivariate time series data in conjunction with the LSTM deep learning technique to forecast the packages' life at different varying temperatures and acceleration levels during vibration.

Download Full-text

Loss-Driven Adversarial Ensemble Deep Learning for On-Line Time Series Analysis

Sustainability ◽

10.3390/su11123489 ◽

2019 ◽

Vol 11 (12) ◽

pp. 3489

Author(s):

Hyungjin Ko ◽

Jaewook Lee ◽

Junyoung Byun ◽

Bumho Son ◽

Saerom Park

Keyword(s):

Time Series ◽

Deep Learning ◽

Time Series Data ◽

Learning Algorithm ◽

Data Distribution ◽

Series Data ◽

Learning Models ◽

Classification Problems ◽

Ensemble Models ◽

On Line

Developing a robust and sustainable system is an important problem in which deep learning models are used in real-world applications. Ensemble methods combine diverse models to improve performance and achieve robustness. The analysis of time series data requires dealing with continuously incoming instances; however, most ensemble models suffer when adapting to a change in data distribution. Therefore, we propose an on-line ensemble deep learning algorithm that aggregates deep learning models and adjusts the ensemble weight based on loss value in this study. We theoretically demonstrate that the ensemble weight converges to the limiting distribution, and, thus, minimizes the average total loss from a new regret measure based on adversarial assumption. We also present an overall framework that can be applied to analyze time series. In the experiments, we focused on the on-line phase, in which the ensemble models predict the binary class for the simulated data and the financial and non-financial real data. The proposed method outperformed other ensemble approaches. Moreover, our method was not only robust to the intentional attacks but also sustainable in data distribution changes. In the future, our algorithm can be extended to regression and multiclass classification problems.

Download Full-text

TapNet: Multivariate Time Series Classification with Attentional Prototypical Network

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i04.6165 ◽

2020 ◽

Vol 34 (04) ◽

pp. 6845-6852 ◽

Cited By ~ 3

Author(s):

Xuchao Zhang ◽

Yifeng Gao ◽

Jessica Lin ◽

Chang-Tien Lu

Keyword(s):

Time Series ◽

Deep Learning ◽

Time Series Data ◽

Multivariate Time Series ◽

Feature Representation ◽

Series Data ◽

Time Series Classification ◽

Random Group ◽

Low Dimensional ◽

Low Dimensional Features

With the advance of sensor technologies, the Multivariate Time Series classification (MTSC) problem, perhaps one of the most essential problems in the time series data mining domain, has continuously received a significant amount of attention in recent decades. Traditional time series classification approaches based on Bag-of-Patterns or Time Series Shapelet have difficulty dealing with the huge amounts of feature candidates generated in high-dimensional multivariate data but have promising performance even when the training set is small. In contrast, deep learning based methods can learn low-dimensional features efficiently but suffer from a shortage of labelled data. In this paper, we propose a novel MTSC model with an attentional prototype network to take the strengths of both traditional and deep learning based approaches. Specifically, we design a random group permutation method combined with multi-layer convolutional networks to learn the low-dimensional features from multivariate time series data. To handle the issue of limited training labels, we propose a novel attentional prototype network to train the feature representation based on their distance to class prototypes with inadequate data labels. In addition, we extend our model into its semi-supervised setting by utilizing the unlabeled data. Extensive experiments on 18 datasets in a public UEA Multivariate time series archive with eight state-of-the-art baseline methods exhibit the effectiveness of the proposed model.

Download Full-text

Prediction of Seoul House Price Index Using Deep Learning Algorithms with Multivariate Time Series Data

SH Urban Research & Insight ◽

10.26700/shuri.2018.08.8.2.39 ◽

2018 ◽

Vol 8 (2) ◽

pp. 39-56 ◽

Cited By ~ 1

Author(s):

Tae Hyeong Lee ◽

Myung-Jin Jun

Keyword(s):

Time Series ◽

Deep Learning ◽

Price Index ◽

Time Series Data ◽

Multivariate Time Series ◽

Learning Algorithms ◽

House Price ◽

Series Data ◽

House Price Index

Download Full-text

A new weighted MSE loss for wind speed forecasting based on deep learning models

10.5194/egusphere-egu2020-12899 ◽

2020 ◽

Author(s):

Xi Chen ◽

Ruyi Yu ◽

Sajid Ullah ◽

Dianming Wu ◽

Min Liu ◽

...

Keyword(s):

Time Series ◽

Deep Learning ◽

Wind Speed ◽

Loss Function ◽

Time Series Data ◽

Series Data ◽

Learning Models ◽

Wind Speed Forecasting ◽

Training Samples ◽

Forecasting Performance

Wind speed forecasting is very important for a lot of real-life applications, especially for controlling and monitoring of wind power plants. Owing to the non-linearity of wind speed time series, it is hard to improve the accuracy of runoff forecasting, especially several days ahead. In order to improve the forecasting performance, many forecasting models have been proposed. Recently, deep learning models have been paid great attention, since they excel the conventional machine learning models. The majority of existing deep learning models take the mean squared error (MSE) loss as the loss function for forecasting. MSE loss is linear. Consequently, it hinders further improvement of forecasting performance over nonlinear wind speed time series data. &#160;&#160; &#160; In this work, we propose a new weighted MSE loss function for wind speed forecasting based on deep learning. As is well known, the training procedure is dominated by easy-training samples in applications. The domination will cause the ineffectiveness and inefficiency of computation. In the new weighted MSE loss function, loss weights of samples can be automatically reduced, according to the contribution of easy-training samples. Thus, the total loss mainly focuses on hard-training samples. To verify the new loss function, Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have been used as base models.&#160; &#160; A number of experiments have been carried out by using open wind speed time series data collected from China and Unites states to demonstrate the effectiveness of the new loss function with three popular models. The performances of the models have been evaluated through the statistical error measures, such as Mean Absolute Error (MAE). MAE of the proposed weighted MSE loss are at most 55% lower than traditional MSE loss. The experimental results indicate that the new weighted loss function can outperform the popular MSE loss function in wind speed forecasting.&#160;

Download Full-text