Evolutionary optimization of long short-term memory neural network language model

2016 ◽  
Vol 140 (4) ◽  
pp. 3062-3062
Author(s):  
Tomohiro Tanaka ◽  
Takafumi Moriya ◽  
Takahiro Shinozaki ◽  
Shinji Watanabe ◽  
Takaaki Hori ◽  
...  
Keyword(s):  
Neural Network ◽  
Short Term Memory ◽  
Language Model ◽  
Evolutionary Optimization ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory ◽  
Network Language

scholarly journals Language Model Using Neural Turing Machine Based on Localized Content-Based Addressing

2020 ◽  
Vol 10 (20) ◽  
pp. 7181
Author(s):  
Donghyun Lee ◽  
Jeong-Sik Park ◽  
Myoung-Wan Koo ◽  
Ji-Hwan Kim
Keyword(s):  
Neural Network ◽  
Turing Machine ◽  
Short Term Memory ◽  
Language Model ◽  
Short Term ◽  
Term Memory ◽  
Memory Address ◽  
Specific Memory ◽  
Memory Area ◽  
Long Short Term Memory

The performance of a long short-term memory (LSTM) recurrent neural network (RNN)-based language model has been improved on language model benchmarks. Although a recurrent layer has been widely used, previous studies showed that an LSTM RNN-based language model (LM) cannot overcome the limitation of the context length. To train LMs on longer sequences, attention mechanism-based models have recently been used. In this paper, we propose a LM using a neural Turing machine (NTM) architecture based on localized content-based addressing (LCA). The NTM architecture is one of the attention-based model. However, the NTM encounters a problem with content-based addressing because all memory addresses need to be accessed for calculating cosine similarities. To address this problem, we propose an LCA method. The LCA method searches for the maximum of all cosine similarities generated from all memory addresses. Next, a specific memory area including the selected memory address is normalized with the softmax function. The LCA method is applied to pre-trained NTM-based LM during the test stage. The proposed architecture is evaluated on Penn Treebank and enwik8 LM tasks. The experimental results indicate that the proposed approach outperforms the previous NTM architecture.

scholarly journals Electricity Consumption Forecasting Based on a Bidirectional Long-Short-Term Memory Artificial Neural Network

2020 ◽  
Vol 13 (1) ◽  
pp. 104
Author(s):  
Dana-Mihaela Petroșanu ◽  
Alexandru Pîrjan
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Short Term Memory ◽  
Electricity Consumption ◽  
Short Term ◽  
Term Memory ◽  
Storage Room ◽  
Long Short Term Memory ◽  
Commercial Center ◽  
Business And Management

The accurate forecasting of the hourly month-ahead electricity consumption represents a very important aspect for non-household electricity consumers and system operators, and at the same time represents a key factor in what regards energy efficiency and achieving sustainable economic, business, and management operations. In this context, we have devised, developed, and validated within the paper an hourly month ahead electricity consumption forecasting method. This method is based on a bidirectional long-short-term memory (BiLSTM) artificial neural network (ANN) enhanced with a multiple simultaneously decreasing delays approach coupled with function fitting neural networks (FITNETs). The developed method targets the hourly month-ahead total electricity consumption at the level of a commercial center-type consumer and for the hourly month ahead consumption of its refrigerator storage room. The developed approach offers excellent forecasting results, highlighted by the validation stage’s results along with the registered performance metrics, namely 0.0495 for the root mean square error (RMSE) performance metric for the total hourly month-ahead electricity consumption and 0.0284 for the refrigerator storage room. We aimed for and managed to attain an hourly month-ahead consumed electricity prediction without experiencing a significant drop in the forecasting accuracy that usually tends to occur after the first two weeks, therefore achieving a reliable method that satisfies the contractor’s needs, being able to enhance his/her activity from the economic, business, and management perspectives. Even if the devised, developed, and validated forecasting solution for the hourly consumption targets a commercial center-type consumer, based on its accuracy, this solution can also represent a useful tool for other non-household electricity consumers due to its generalization capability.

Sign in / Sign up

Export Citation Format

Share Document