SACNN: Self-attentive Convolutional Neural Network Model for Natural Language Inference

2021 ◽  
Vol 20 (3) ◽  
pp. 1-16
Author(s):  
Waris Quamer ◽  
Praphula Kumar Jain ◽  
Arpit Rai ◽  
Vijayalakshmi Saravanan ◽  
Rajendra Pamula ◽  
...  
Keyword(s):  
Neural Network ◽  
Natural Language ◽  
Short Term Memory ◽  
Local Context ◽  
Short Term ◽  
Proposed Model ◽  
Long Short Term Memory ◽  
Complex Relationships ◽  
Gated Recurrent Units

Inference has been central problem for understanding and reasoning in artificial intelligence. Especially, Natural Language Inference is an interesting problem that has attracted the attention of many researchers. Natural language inference intends to predict whether a hypothesis sentence can be inferred from the premise sentence. Most prior works rely on a simplistic association between the premise and hypothesis sentence pairs, which is not sufficient for learning complex relationships between them. The strategy also fails to exploit local context information fully. Long Short Term Memory (LSTM) or gated recurrent units networks (GRU) are not effective in modeling long-term dependencies, and their schemes are far more complex as compared to Convolutional Neural Networks (CNN). To address this problem of long-term dependency, and to involve context for modeling better representation of a sentence, in this article, a general Self-Attentive Convolution Neural Network (SACNN) is presented for natural language inference and sentence pair modeling tasks. The proposed model uses CNNs to integrate mutual interactions between sentences, and each sentence with their counterparts is taken into consideration for the formulation of their representation. Moreover, the self-attention mechanism helps fully exploit the context semantics and long-term dependencies within a sentence. Experimental results proved that SACNN was able to outperform strong baselines and achieved an accuracy of 89.7% on the stanford natural language inference (SNLI) dataset.

Sentence similarity evaluation using Sent2Vec and siamese neural network with parallel structure

2021 ◽  
pp. 1-10
Author(s):  
Hye-Jeong Song ◽  
Tak-Sung Heo ◽  
Jong-Dae Kim ◽  
Chan-Young Park ◽  
Yu-Seop Kim
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Short Term Memory ◽  
Parallel Structure ◽  
Short Term ◽  
Similarity Estimation ◽  
Accurate Judgment ◽  
Proposed Model ◽  
Sentence Similarity ◽  
Long Short Term Memory

Sentence similarity evaluation is a significant task used in machine translation, classification, and information extraction in the field of natural language processing. When two sentences are given, an accurate judgment should be made whether the meaning of the sentences is equivalent even if the words and contexts of the sentences are different. To this end, existing studies have measured the similarity of sentences by focusing on the analysis of words, morphemes, and letters. To measure sentence similarity, this study uses Sent2Vec, a sentence embedding, as well as morpheme word embedding. Vectors representing words are input to the 1-dimension convolutional neural network (1D-CNN) with various sizes of kernels and bidirectional long short-term memory (Bi-LSTM). Self-attention is applied to the features transformed through Bi-LSTM. Subsequently, vectors undergoing 1D-CNN and self-attention are converted through global max pooling and global average pooling to extract specific values, respectively. The vectors generated through the above process are concatenated to the vector generated through Sent2Vec and are represented as a single vector. The vector is input to softmax layer, and finally, the similarity between the two sentences is determined. The proposed model can improve the accuracy by up to 5.42% point compared with the conventional sentence similarity estimation models.

scholarly journals Natural language description of images using hybrid recurrent neural network

2019 ◽  
Vol 9 (4) ◽  
pp. 2932
Author(s):  
Md. Asifuzzaman Jishan ◽  
Khan Raqib Mahmud ◽  
Abul Kalam Al Azad
Keyword(s):  
Neural Network ◽  
Natural Language ◽  
Recurrent Neural Network ◽  
Short Term Memory ◽  
Text Line ◽  
Short Term ◽  
Word Representation ◽  
Benchmark Datasets ◽  
Long Short Term Memory ◽  
Language Description

We presented a learning model that generated natural language description of images. The model utilized the connections between natural language and visual data by produced text line based contents from a given image. Our Hybrid Recurrent Neural Network model is based on the intricacies of Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Bi-directional Recurrent Neural Network (BRNN) models. We conducted experiments on three benchmark datasets, e.g., Flickr8K, Flickr30K, and MS COCO. Our hybrid model utilized LSTM model to encode text line or sentences independent of the object location and BRNN for word representation, this reduced the computational complexities without compromising the accuracy of the descriptor. The model produced better accuracy in retrieving natural language based description on the dataset.

Production Forecasting with the Interwell Interference by Integrating Graph Convolutional and Long Short-Term Memory Neural Network

2021 ◽  
pp. 1-17
Author(s):  
Enda Du ◽  
Yuetian Liu ◽  
Ziyan Cheng ◽  
Liang Xue ◽  
Jing Ma ◽  
...  
Keyword(s):  
Neural Network ◽  
Short Term Memory ◽  
Short Term ◽  
Term Memory ◽  
Production Forecasting ◽  
Temporal Correlations ◽  
Proposed Model ◽  
The Mean ◽  
Long Short Term Memory ◽  
The Impact

Summary Accurate production forecasting is an essential task and accompanies the entire process of reservoir development. With the limitation of prediction principles and processes, the traditional approaches are difficult to make rapid predictions. With the development of artificial intelligence, the data-driven model provides an alternative approach for production forecasting. To fully take the impact of interwell interference on production into account, this paper proposes a deep learning-based hybrid model (GCN-LSTM), where graph convolutional network (GCN) is used to capture complicated spatial patterns between each well, and long short-term memory (LSTM) neural network is adopted to extract intricate temporal correlations from historical production data. To implement the proposed model more efficiently, two data preprocessing procedures are performed: Outliers in the data set are removed by using a box plot visualization, and measurement noise is reduced by a wavelet transform. The robustness and applicability of the proposed model are evaluated in two scenarios of different data types with the root mean square error (RMSE), the mean absolute error (MAE), and the mean absolute percentage error (MAPE). The results show that the proposed model can effectively capture spatial and temporal correlations to make a rapid and accurate oil production forecast.

Categorizing Natural Language-Based Customer Satisfaction: An Implementation Method Using Support Vector Machine and Long Short-Term Memory Neural Network

2021 ◽  
Vol 13 (4) ◽  
Author(s):  
Ralph Sherwin A. Corpuz ◽  
Keyword(s):  
Neural Network ◽  
Support Vector Machine ◽  
Natural Language ◽  
Text Categorization ◽  
Short Term Memory ◽  
Support Vector ◽  
Feature Engineering ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory

Analyzing natural language-based Customer Satisfaction (CS) is a tedious process. This issue is practically true if one is to manually categorize large datasets. Fortunately, the advent of supervised machine learning techniques has paved the way toward the design of efficient categorization systems used for CS. This paper presents the feasibility of designing a text categorization model using two popular and robust algorithms – the Support Vector Machine (SVM) and Long Short-Term Memory (LSTM) Neural Network, in order to automatically categorize complaints, suggestions, feedbacks, and commendations. The study found that, in terms of training accuracy, SVM has best rating of 98.63% while LSTM has best rating of 99.32%. Such results mean that both SVM and LSTM algorithms are at par with each other in terms of training accuracy, but SVM is significantly faster than LSTM by approximately 35.47s. The training performance results of both algorithms are attributed on the limitations of the dataset size, high-dimensionality of both English and Tagalog languages, and applicability of the feature engineering techniques used. Interestingly, based on the results of actual implementation, both algorithms are found to be 100% effective in accurately predicting the correct CS categories. Hence, the extent of preference between the two algorithms boils down on the available dataset and the skill in optimizing these algorithms through feature engineering techniques and in implementing them toward actual text categorization applications.

scholarly journals Long Short Term Memory Recurrent Network for Standard and Poor’s 500 Index Modelling

2018 ◽  
Vol 7 (4.15) ◽  
pp. 25 ◽  
Author(s):  
Said Jadid Abdulkadir ◽  
Hitham Alhussian ◽  
Muhammad Nazmi ◽  
Asim A Elsheikh
Keyword(s):  
Neural Network ◽  
Time Series ◽  
Recurrent Neural Network ◽  
Time Series Data ◽  
Short Term Memory ◽  
Series Data ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory

Forecasting time-series data are imperative especially when planning is required through modelling using uncertain knowledge of future events. Recurrent neural network models have been applied in the industry and outperform standard artificial neural networks in forecasting, but fail in long term time-series forecasting due to the vanishing gradient problem. This study offers a robust solution that can be implemented for long-term forecasting using a special architecture of recurrent neural network known as Long Short Term Memory (LSTM) model to overcome the vanishing gradient problem. LSTM is specially designed to avoid the long-term dependency problem as their default behavior. Empirical analysis is performed using quantitative forecasting metrics and comparative model performance on the forecasted outputs. An evaluation analysis is performed to validate that the LSTM model provides better forecasted outputs on Standard & Poor’s 500 Index (S&P 500) in terms of error metrics as compared to other forecasting models.  

scholarly journals Forecasting Day-Ahead Hourly Photovoltaic Power Generation Using Convolutional Self-Attention Based Long Short-Term Memory

Energies ◽  
2020 ◽  
Vol 13 (15) ◽  
pp. 4017 ◽  
Author(s):  
Dukhwan Yu ◽  
Wonik Choi ◽  
Myoungsoo Kim ◽  
Ling Liu
Keyword(s):  
Power Generation ◽  
Deep Learning ◽  
Prediction Accuracy ◽  
Short Term Memory ◽  
Local Context ◽  
Short Term ◽  
Term Memory ◽  
Proposed Model ◽  
Long Short Term Memory ◽  
Pv Power Generation

The problem of Photovoltaic (PV) power generation forecasting is becoming crucial as the penetration level of Distributed Energy Resources (DERs) increases in microgrids and Virtual Power Plants (VPPs). In order to improve the stability of power systems, a fair amount of research has been proposed for increasing prediction performance in practical environments through statistical, machine learning, deep learning, and hybrid approaches. Despite these efforts, the problem of forecasting PV power generation remains to be challenging in power system operations since existing methods show limited accuracy and thus are not sufficiently practical enough to be widely deployed. Many existing methods using long historical data suffer from the long-term dependency problem and are not able to produce high prediction accuracy due to their failure to fully utilize all features of long sequence inputs. To address this problem, we propose a deep learning-based PV power generation forecasting model called Convolutional Self-Attention based Long Short-Term Memory (LSTM). By using the convolutional self-attention mechanism, we can significantly improve prediction accuracy by capturing the local context of the data and generating keys and queries that fit the local context. To validate the applicability of the proposed model, we conduct extensive experiments on both PV power generation forecasting using a real world dataset and power consumption forecasting. The experimental results of power generation forecasting using the real world datasets show that the MAPEs of the proposed model are much lower, in fact by 7.7%, 6%, 3.9% compared to the Deep Neural Network (DNN), LSTM and LSTM with the canonical self-attention, respectively. As for power consumption forecasting, the proposed model exhibits 32%, 17% and 44% lower Mean Absolute Percentage Error (MAPE) than the DNN, LSTM and LSTM with the canonical self-attention, respectively.

scholarly journals An LSTM-Based Method with Attention Mechanism for Travel Time Prediction

Sensors ◽  
2019 ◽  
Vol 19 (4) ◽  
pp. 861 ◽  
Author(s):  
Xiangdong Ran ◽  
Zhiguang Shan ◽  
Yufei Fang ◽  
Chuang Lin
Keyword(s):  
Short Term Memory ◽  
Attention Mechanism ◽  
Traffic Prediction ◽  
Travel Time Prediction ◽  
Short Term ◽  
Term Memory ◽  
Proposed Model ◽  
Departure Time ◽  
Long Short Term Memory

Traffic prediction is based on modeling the complex non-linear spatiotemporal traffic dynamics in road network. In recent years, Long Short-Term Memory has been applied to traffic prediction, achieving better performance. The existing Long Short-Term Memory methods for traffic prediction have two drawbacks: they do not use the departure time through the links for traffic prediction, and the way of modeling long-term dependence in time series is not direct in terms of traffic prediction. Attention mechanism is implemented by constructing a neural network according to its task and has recently demonstrated success in a wide range of tasks. In this paper, we propose an Long Short-Term Memory-based method with attention mechanism for travel time prediction. We present the proposed model in a tree structure. The proposed model substitutes a tree structure with attention mechanism for the unfold way of standard Long Short-Term Memory to construct the depth of Long Short-Term Memory and modeling long-term dependence. The attention mechanism is over the output layer of each Long Short-Term Memory unit. The departure time is used as the aspect of the attention mechanism and the attention mechanism integrates departure time into the proposed model. We use AdaGrad method for training the proposed model. Based on the datasets provided by Highways England, the experimental results show that the proposed model can achieve better accuracy than the Long Short-Term Memory and other baseline methods. The case study suggests that the departure time is effectively employed by using attention mechanism.

scholarly journals Long Short-Term Memory with Dynamic Skip Connections

2019 ◽  
Vol 33 ◽  
pp. 6481-6488 ◽  
Author(s):  
Tao Gui ◽  
Qi Zhang ◽  
Lujun Zhao ◽  
Yaosong Lin ◽  
Minlong Peng ◽  
...  
Keyword(s):  
Language Processing ◽  
Short Term Memory ◽  
Training Data ◽  
Sequential Data ◽  
Short Term ◽  
Term Memory ◽  
Transition Functions ◽  
Proposed Model ◽  
Long Short Term Memory

In recent years, long short-term memory (LSTM) has been successfully used to model sequential data of variable length. However, LSTM can still experience difficulty in capturing long-term dependencies. In this work, we tried to alleviate this problem by introducing a dynamic skip connection, which can learn to directly connect two dependent words. Since there is no dependency information in the training data, we propose a novel reinforcement learning-based method to model the dependency relationship and connect dependent words. The proposed model computes the recurrent transition functions based on the skip connections, which provides a dynamic skipping advantage over RNNs that always tackle entire sentences sequentially. Our experimental results on three natural language processing tasks demonstrate that the proposed method can achieve better performance than existing methods. In the number prediction experiment, the proposed model outperformed LSTM with respect to accuracy by nearly 20%.

scholarly journals Crop Yield Estimation Using Deep Learning Based on Climate Big Data and Irrigation Scheduling

Energies ◽  
2021 ◽  
Vol 14 (11) ◽  
pp. 3004
Author(s):  
Khadijeh Alibabaei ◽  
Pedro D. Gaspar ◽  
Tânia M. Lima
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Energy Consumption ◽  
Short Term Memory ◽  
Irrigation Scheduling ◽  
Climate Data ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory ◽  
Gated Recurrent Units

Deep learning has already been successfully used in the development of decision support systems in various domains. Therefore, there is an incentive to apply it in other important domains such as agriculture. Fertilizers, electricity, chemicals, human labor, and water are the components of total energy consumption in agriculture. Yield estimates are critical for food security, crop management, irrigation scheduling, and estimating labor requirements for harvesting and storage. Therefore, estimating product yield can reduce energy consumption. Two deep learning models, Long Short-Term Memory and Gated Recurrent Units, have been developed for the analysis of time-series data such as agricultural datasets. In this paper, the capabilities of these models and their extensions, called Bidirectional Long Short-Term Memory and Bidirectional Gated Recurrent Units, to predict end-of-season yields are investigated. The models use historical data, including climate data, irrigation scheduling, and soil water content, to estimate end-of-season yield. The application of this technique was tested for tomato and potato yields at a site in Portugal. The Bidirectional Long Short-Term memory outperformed the Gated Recurrent Units network, the Long Short-Term Memory, and the Bidirectional Gated Recurrent Units network on the validation dataset. The model was able to capture the nonlinear relationship between irrigation amount, climate data, and soil water content and predict yield with an MSE of 0.017 to 0.039. The performance of the Bidirectional Long Short-Term Memory in the test was compared with the most commonly used deep learning method, the Convolutional Neural Network, and machine learning methods including a Multi-Layer Perceptrons model and Random Forest Regression. The Bidirectional Long Short-Term Memory outperformed the other models with an R2 score between 0.97 and 0.99. The results show that analyzing agricultural data with the Long Short-Term Memory model improves the performance of the model in terms of accuracy. The Convolutional Neural Network model achieved the second-best performance. Therefore, the deep learning model has a remarkable ability to predict the yield at the end of the season.

Sign in / Sign up

Export Citation Format

Share Document