Neuroevolution of a Modular Memory-Augmented Neural Network for Deep Memory Problems

2019 ◽  
Vol 27 (4) ◽  
pp. 639-664 ◽  
Author(s):  
Shauharda Khadka ◽  
Jen Jen Chung ◽  
Kagan Tumer
Keyword(s):  
Neural Network ◽  
Gradient Descent ◽  
Short Term Memory ◽  
Extended Period ◽  
New Class ◽  
Neural Architecture ◽  
Memory Block ◽  
Memory Problems ◽  
Long Short Term Memory ◽  
Gated Recurrent Units

We present Modular Memory Units (MMUs), a new class of memory-augmented neural network. MMU builds on the gated neural architecture of Gated Recurrent Units (GRUs) and Long Short Term Memory (LSTMs), to incorporate an external memory block, similar to a Neural Turing Machine (NTM). MMU interacts with the memory block using independent read and write gates that serve to decouple the memory from the central feedforward operation. This allows for regimented memory access and update, giving our network the ability to choose when to read from memory, update it, or simply ignore it. This capacity to act in detachment allows the network to shield the memory from noise and other distractions, while simultaneously using it to effectively retain and propagate information over an extended period of time. We train MMU using both neuroevolution and gradient descent, and perform experiments on two deep memory benchmarks. Results demonstrate that MMU performs significantly faster and more accurately than traditional LSTM-based methods, and is robust to dramatic increases in the sequence depth of these memory benchmarks.

scholarly journals Crop Yield Estimation Using Deep Learning Based on Climate Big Data and Irrigation Scheduling

Energies ◽  
2021 ◽  
Vol 14 (11) ◽  
pp. 3004
Author(s):  
Khadijeh Alibabaei ◽  
Pedro D. Gaspar ◽  
Tânia M. Lima
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Energy Consumption ◽  
Short Term Memory ◽  
Irrigation Scheduling ◽  
Climate Data ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory ◽  
Gated Recurrent Units

Deep learning has already been successfully used in the development of decision support systems in various domains. Therefore, there is an incentive to apply it in other important domains such as agriculture. Fertilizers, electricity, chemicals, human labor, and water are the components of total energy consumption in agriculture. Yield estimates are critical for food security, crop management, irrigation scheduling, and estimating labor requirements for harvesting and storage. Therefore, estimating product yield can reduce energy consumption. Two deep learning models, Long Short-Term Memory and Gated Recurrent Units, have been developed for the analysis of time-series data such as agricultural datasets. In this paper, the capabilities of these models and their extensions, called Bidirectional Long Short-Term Memory and Bidirectional Gated Recurrent Units, to predict end-of-season yields are investigated. The models use historical data, including climate data, irrigation scheduling, and soil water content, to estimate end-of-season yield. The application of this technique was tested for tomato and potato yields at a site in Portugal. The Bidirectional Long Short-Term memory outperformed the Gated Recurrent Units network, the Long Short-Term Memory, and the Bidirectional Gated Recurrent Units network on the validation dataset. The model was able to capture the nonlinear relationship between irrigation amount, climate data, and soil water content and predict yield with an MSE of 0.017 to 0.039. The performance of the Bidirectional Long Short-Term Memory in the test was compared with the most commonly used deep learning method, the Convolutional Neural Network, and machine learning methods including a Multi-Layer Perceptrons model and Random Forest Regression. The Bidirectional Long Short-Term Memory outperformed the other models with an R2 score between 0.97 and 0.99. The results show that analyzing agricultural data with the Long Short-Term Memory model improves the performance of the model in terms of accuracy. The Convolutional Neural Network model achieved the second-best performance. Therefore, the deep learning model has a remarkable ability to predict the yield at the end of the season.

SACNN: Self-attentive Convolutional Neural Network Model for Natural Language Inference

2021 ◽  
Vol 20 (3) ◽  
pp. 1-16
Author(s):  
Waris Quamer ◽  
Praphula Kumar Jain ◽  
Arpit Rai ◽  
Vijayalakshmi Saravanan ◽  
Rajendra Pamula ◽  
...  
Keyword(s):  
Neural Network ◽  
Natural Language ◽  
Short Term Memory ◽  
Local Context ◽  
Short Term ◽  
Proposed Model ◽  
Long Short Term Memory ◽  
Complex Relationships ◽  
Gated Recurrent Units

Inference has been central problem for understanding and reasoning in artificial intelligence. Especially, Natural Language Inference is an interesting problem that has attracted the attention of many researchers. Natural language inference intends to predict whether a hypothesis sentence can be inferred from the premise sentence. Most prior works rely on a simplistic association between the premise and hypothesis sentence pairs, which is not sufficient for learning complex relationships between them. The strategy also fails to exploit local context information fully. Long Short Term Memory (LSTM) or gated recurrent units networks (GRU) are not effective in modeling long-term dependencies, and their schemes are far more complex as compared to Convolutional Neural Networks (CNN). To address this problem of long-term dependency, and to involve context for modeling better representation of a sentence, in this article, a general Self-Attentive Convolution Neural Network (SACNN) is presented for natural language inference and sentence pair modeling tasks. The proposed model uses CNNs to integrate mutual interactions between sentences, and each sentence with their counterparts is taken into consideration for the formulation of their representation. Moreover, the self-attention mechanism helps fully exploit the context semantics and long-term dependencies within a sentence. Experimental results proved that SACNN was able to outperform strong baselines and achieved an accuracy of 89.7% on the stanford natural language inference (SNLI) dataset.

scholarly journals Electricity Consumption Forecasting Based on a Bidirectional Long-Short-Term Memory Artificial Neural Network

2020 ◽  
Vol 13 (1) ◽  
pp. 104
Author(s):  
Dana-Mihaela Petroșanu ◽  
Alexandru Pîrjan
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Short Term Memory ◽  
Electricity Consumption ◽  
Short Term ◽  
Term Memory ◽  
Storage Room ◽  
Long Short Term Memory ◽  
Commercial Center ◽  
Business And Management

The accurate forecasting of the hourly month-ahead electricity consumption represents a very important aspect for non-household electricity consumers and system operators, and at the same time represents a key factor in what regards energy efficiency and achieving sustainable economic, business, and management operations. In this context, we have devised, developed, and validated within the paper an hourly month ahead electricity consumption forecasting method. This method is based on a bidirectional long-short-term memory (BiLSTM) artificial neural network (ANN) enhanced with a multiple simultaneously decreasing delays approach coupled with function fitting neural networks (FITNETs). The developed method targets the hourly month-ahead total electricity consumption at the level of a commercial center-type consumer and for the hourly month ahead consumption of its refrigerator storage room. The developed approach offers excellent forecasting results, highlighted by the validation stage’s results along with the registered performance metrics, namely 0.0495 for the root mean square error (RMSE) performance metric for the total hourly month-ahead electricity consumption and 0.0284 for the refrigerator storage room. We aimed for and managed to attain an hourly month-ahead consumed electricity prediction without experiencing a significant drop in the forecasting accuracy that usually tends to occur after the first two weeks, therefore achieving a reliable method that satisfies the contractor’s needs, being able to enhance his/her activity from the economic, business, and management perspectives. Even if the devised, developed, and validated forecasting solution for the hourly consumption targets a commercial center-type consumer, based on its accuracy, this solution can also represent a useful tool for other non-household electricity consumers due to its generalization capability.

Sign in / Sign up

Export Citation Format

Share Document