scholarly journals Source Code Assessment and Classification Based on Estimated Error Probability Using Attentive LSTM Language Model and Its Application in Programming Education

2020 ◽  
Vol 10 (8) ◽  
pp. 2973 ◽  
Author(s):  
Md. Mostafizer Rahman ◽  
Yutaka Watanobe ◽  
Keita Nakamura
Keyword(s):  
Error Detection ◽  
Error Probability ◽  
Short Term Memory ◽  
Language Model ◽  
Source Code ◽  
Attention Mechanism ◽  
Error Assessment ◽  
Programming Education ◽  
Proposed Model ◽  
Estimated Error

The rate of software development has increased dramatically. Conventional compilers cannot assess and detect all source code errors. Software may thus contain errors, negatively affecting end-users. It is also difficult to assess and detect source code logic errors using traditional compilers, resulting in software that contains errors. A method that utilizes artificial intelligence for assessing and detecting errors and classifying source code as correct (error-free) or incorrect is thus required. Here, we propose a sequential language model that uses an attention-mechanism-based long short-term memory (LSTM) neural network to assess and classify source code based on the estimated error probability. The attentive mechanism enhances the accuracy of the proposed language model for error assessment and classification. We trained the proposed model using correct source code and then evaluated its performance. The experimental results show that the proposed model has logic and syntax error detection accuracies of 92.2% and 94.8%, respectively, outperforming state-of-the-art models. We also applied the proposed model to the classification of source code with logic and syntax errors. The average precision, recall, and F-measure values for such classification are much better than those of benchmark models. To strengthen the proposed model, we combined the attention mechanism with LSTM to enhance the results of error assessment and detection as well as source code classification. Finally, our proposed model can be effective in programming education and software engineering by improving code writing, debugging, error-correction, and reasoning.

scholarly journals A Bidirectional LSTM Language Model for Code Evaluation and Repair

Symmetry ◽  
2021 ◽  
Vol 13 (2) ◽  
pp. 247
Author(s):  
Md. Mostafizer Rahman ◽  
Yutaka Watanobe ◽  
Keita Nakamura
Keyword(s):  
Error Detection ◽  
Short Term Memory ◽  
Language Model ◽  
Source Code ◽  
Short Term ◽  
Science And Engineering ◽  
Term Memory ◽  
Source Codes ◽  
Long Short Term Memory ◽  
Logical Errors

Programming is a vital skill in computer science and engineering-related disciplines. However, developing source code is an error-prone task. Logical errors in code are particularly hard to identify for both students and professionals, and a single error is unexpected to end-users. At present, conventional compilers have difficulty identifying many of the errors (especially logical errors) that can occur in code. To mitigate this problem, we propose a language model for evaluating source codes using a bidirectional long short-term memory (BiLSTM) neural network. We trained the BiLSTM model with a large number of source codes with tuning various hyperparameters. We then used the model to evaluate incorrect code and assessed the model’s performance in three principal areas: source code error detection, suggestions for incorrect code repair, and erroneous code classification. Experimental results showed that the proposed BiLSTM model achieved 50.88% correctness in identifying errors and providing suggestions. Moreover, the model achieved an F-score of approximately 97%, outperforming other state-of-the-art models (recurrent neural networks (RNNs) and long short-term memory (LSTM)).

Improve Language Modeling for Code Completion Through Learning General Token Repetition of Source Code with Optimized Memory

2019 ◽  
Vol 29 (11n12) ◽  
pp. 1801-1818
Author(s):  
Yixiao Yang ◽  
Xiang Chen ◽  
Jiaguang Sun
Keyword(s):  
Prediction Accuracy ◽  
Language Model ◽  
Source Code ◽  
Attention Mechanism ◽  
Data Set ◽  
Training Time ◽  
Unseen Data ◽  
Code Completion ◽  
High Chance ◽  
Better Than

In last few years, applying language model to source code is the state-of-the-art method for solving the problem of code completion. However, compared with natural language, code has more obvious repetition characteristics. For example, a variable can be used many times in the following code. Variables in source code have a high chance to be repetitive. Cloned code and templates, also have the property of token repetition. Capturing the token repetition of source code is important. In different projects, variables or types are usually named differently. This means that a model trained in a finite data set will encounter a lot of unseen variables or types in another data set. How to model the semantics of the unseen data and how to predict the unseen data based on the patterns of token repetition are two challenges in code completion. Hence, in this paper, token repetition is modelled as a graph, we propose a novel REP model which is based on deep neural graph network to learn the code toke repetition. The REP model is to identify the edge connections of a graph to recognize the token repetition. For predicting the token repetition of token [Formula: see text], the information of all the previous tokens needs to be considered. We use memory neural network (MNN) to model the semantics of each distinct token to make the framework of REP model more targeted. The experiments indicate that the REP model performs better than LSTM model. Compared with Attention-Pointer network, we also discover that the attention mechanism does not work in all situations. The proposed REP model could achieve similar or slightly better prediction accuracy compared to Attention-Pointer network and consume less training time. We also find other attention mechanism which could further improve the prediction accuracy.

scholarly journals An LSTM-Based Method with Attention Mechanism for Travel Time Prediction

Sensors ◽  
2019 ◽  
Vol 19 (4) ◽  
pp. 861 ◽  
Author(s):  
Xiangdong Ran ◽  
Zhiguang Shan ◽  
Yufei Fang ◽  
Chuang Lin
Keyword(s):  
Short Term Memory ◽  
Attention Mechanism ◽  
Traffic Prediction ◽  
Travel Time Prediction ◽  
Short Term ◽  
Term Memory ◽  
Proposed Model ◽  
Departure Time ◽  
Long Short Term Memory

Traffic prediction is based on modeling the complex non-linear spatiotemporal traffic dynamics in road network. In recent years, Long Short-Term Memory has been applied to traffic prediction, achieving better performance. The existing Long Short-Term Memory methods for traffic prediction have two drawbacks: they do not use the departure time through the links for traffic prediction, and the way of modeling long-term dependence in time series is not direct in terms of traffic prediction. Attention mechanism is implemented by constructing a neural network according to its task and has recently demonstrated success in a wide range of tasks. In this paper, we propose an Long Short-Term Memory-based method with attention mechanism for travel time prediction. We present the proposed model in a tree structure. The proposed model substitutes a tree structure with attention mechanism for the unfold way of standard Long Short-Term Memory to construct the depth of Long Short-Term Memory and modeling long-term dependence. The attention mechanism is over the output layer of each Long Short-Term Memory unit. The departure time is used as the aspect of the attention mechanism and the attention mechanism integrates departure time into the proposed model. We use AdaGrad method for training the proposed model. Based on the datasets provided by Highways England, the experimental results show that the proposed model can achieve better accuracy than the Long Short-Term Memory and other baseline methods. The case study suggests that the departure time is effectively employed by using attention mechanism.

Logic Error Detection Algorithm Based on RNN with Threshold Selection

2020 ◽  
Author(s):  
Taku Matsumoto ◽  
Yutaka Watanobe ◽  
Keita Nakamura ◽  
Yunosuke Teshima
Keyword(s):  
Neural Network ◽  
Recurrent Neural Network ◽  
Error Detection ◽  
Language Model ◽  
Source Code ◽  
Detection Algorithm ◽  
Detection Performance ◽  
Educational Systems ◽  
Determination Method ◽  
Logical Errors

Logical errors in source code can be detected by probabilities obtained from a language model trained by the recurrent neural network (RNN). Using the probabilities and determining thresholds, places that are likely to be logic errors can be enumerated. However, when the threshold is set inappropriately, user may miss true logical errors because of passive extraction or unnecessary elements obtained from excessive extraction. Moreover, the probabilities of output from the language model are different for each task, so the threshold should be selected properly. In this paper, we propose a logic error detection algorithm using an RNN and an automatic threshold determination method. The proposed method selects thresholds using incorrect codes and can enhance the detection performance of the trained language model. For evaluating the proposed method, experiments with data from an online judge system, which is one of the educational systems that provide the automated judge for many programming tasks, are conducted. The experimental results show that the selected thresholds can be used to improve the logic error detection performance of the trained language model.

DEEPSMP: A deep learning model for predicting the ectodomain shedding events of membrane proteins

2020 ◽  
Vol 18 (03) ◽  
pp. 2050017
Author(s):  
Zhongbo Cao ◽  
Wei Du ◽  
Gaoyang Li ◽  
Huansheng Cao
Keyword(s):  
Membrane Proteins ◽  
Cross Validation ◽  
Short Term Memory ◽  
Characteristic Curve ◽  
Attention Mechanism ◽  
Training Dataset ◽  
Sequence Information ◽  
Dense Layer ◽  
Ectodomain Shedding ◽  
Proposed Model

Membrane proteins play essential roles in modern medicine. In recent studies, some membrane proteins involved in ectodomain shedding events have been reported as the potential drug targets and biomarkers of some serious diseases. However, there are few effective tools for identifying the shedding event of membrane proteins. So, it is necessary to design an effective tool for predicting shedding event of membrane proteins. In this study, we design an end-to-end prediction model using deep neural networks with long short-term memory (LSTM) units and attention mechanism, to predict the ectodomain shedding events of membrane proteins only by sequence information. Firstly, the evolutional profiles are encoded from original sequences of these proteins by Position-Specific Iterated BLAST (PSI-BLAST) on Uniref50 database. Then, the LSTM units which contain memory cells are used to hold information from past inputs to the network and the attention mechanism is applied to detect sorting signals in proteins regardless of their position in the sequence. Finally, a fully connected dense layer and a softmax layer are used to obtain the final prediction results. Additionally, we also try to reduce overfitting of the model by using dropout, L2 regularization, and bagging ensemble learning in the model training process. In order to ensure the fairness of performance comparison, firstly we use cross validation process on training dataset obtained from an existing paper. The average accuracy and area under a receiver operating characteristic curve (AUC) of five-fold cross-validation are 81.19% and 0.835 using our proposed model, compared to 75% and 0.78 by a previously published tool, respectively. To better validate the performance of the proposed model, we also evaluate the performance of the proposed model on independent test dataset. The accuracy, sensitivity, and specificity are 83.14%, 84.08%, and 81.63% using our proposed model, compared to 70.20%, 71.97%, and 67.35% by the existing model. The experimental results validate that the proposed model can be regarded as a general tool for predicting ectodomain shedding events of membrane proteins. The pipeline of the model and prediction results can be accessed at the following URL: http://www.csbg-jlu.info/DeepSMP/ .

scholarly journals A Neural Network Based Intelligent Support Model for Program Code Completion

2020 ◽  
Vol 2020 ◽  
pp. 1-18
Author(s):  
Md. Mostafizer Rahman ◽  
Yutaka Watanobe ◽  
Keita Nakamura
Keyword(s):  
Neural Network ◽  
Software Engineering ◽  
Error Detection ◽  
State Of The Art ◽  
Source Code ◽  
Experimental Results ◽  
Intelligent Support ◽  
Source Codes ◽  
Proposed Model ◽  
Code Completion

In recent years, millions of source codes are generated in different languages on a daily basis all over the world. A deep neural network-based intelligent support model for source code completion would be a great advantage in software engineering and programming education fields. Vast numbers of syntax, logical, and other critical errors that cannot be detected by normal compilers continue to exist in source codes, and the development of an intelligent evaluation methodology that does not rely on manual compilation has become essential. Even experienced programmers often find it necessary to analyze an entire program in order to find a single error and are thus being forced to waste valuable time debugging their source codes. With this point in mind, we proposed an intelligent model that is based on long short-term memory (LSTM) and combined it with an attention mechanism for source code completion. Thus, the proposed model can detect source code errors with locations and then predict the correct words. In addition, the proposed model can classify the source codes as to whether they are erroneous or not. We trained our proposed model using the source code and then evaluated the performance. All of the data used in our experiments were extracted from Aizu Online Judge (AOJ) system. The experimental results obtained show that the accuracy in terms of error detection and prediction of our proposed model approximately is 62% and source code classification accuracy is approximately 96% which outperformed a standard LSTM and other state-of-the-art models. Moreover, in comparison to state-of-the-art models, our proposed model achieved an interesting level of success in terms of error detection, prediction, and classification when applied to long source code sequences. Overall, these experimental results indicate the usefulness of our proposed model in software engineering and programming education arena.

Intelligent mining vulnerabilities in python code snippets

2021 ◽  
pp. 1-14
Author(s):  
Wenbo Guo ◽  
Cheng Huang ◽  
Weina Niu ◽  
Yong Fang
Keyword(s):  
Open Source ◽  
Short Term Memory ◽  
Source Code ◽  
Classification Model ◽  
Detection Methods ◽  
Mining System ◽  
Content Description ◽  
Proposed Model ◽  
Neural Network Algorithm ◽  
Multi Classification

In the software development process, many developers learn from code snippets in the open-source community to implement specific functions. However, few people think about whether these code have vulnerabilities, which provides channels for developing unsafe programs. To this end, this paper constructs a source code snippets vulnerability mining system named PyVul based on deep learning to automatically detect the security of code snippets in the open source community. PyVul builds abstract syntax tree (AST) for the source code to extract its code feature, and then introduces the bidirectional long-term short-term memory (BiLSTM) neural network algorithm to detect vulnerability codes. If it is unsafe code, the further constructed a multi-classification model could analyze the context discussion contents in associated threads, to classify the code vulnerability type based the content description. Compared with traditional detection methods, this method can identify unsafe code and classify vulnerability type. The accuracy of the proposed model can reach 85%. PyVul also found 138 vulnerable code snippets in the real public open-source community. In the future, it can be used in the open-source community for vulnerable code auditing to assist users in safe development.

scholarly journals Multi-Layer Attention Approach for Aspect based Sentiment Analysis

2020 ◽  
Author(s):  
Xinzhi Ai ◽  
Xiaoge Li ◽  
Feixiong Hu ◽  
Shuting Zhi ◽  
Likun Hu
Keyword(s):  
Sentiment Analysis ◽  
Semantic Information ◽  
Short Term Memory ◽  
Attention Mechanism ◽  
Training Dataset ◽  
Emotion Classification ◽  
Data Set ◽  
New Model ◽  
Fine Grained ◽  
Proposed Model

Based on the aspect-level sentiment analysis is typical of fine-grained emotional classification that assigns sentiment polarity for each of the aspects in a review. For better handle the emotion classification task, this paper put forward a new model which apply Long Short-Term Memory network combine multiple attention with aspect context. Where multiple attention mechanism (i.e., location attention, content attention and class attention) refers to takes the factors of context location, content semantics and class balancing into consideration. Therefore, the proposed model can adaptively integrate location and semantic information between the aspect targets and their contexts into sentimental features, and overcome the model data variance introduced by the imbalanced training dataset. In addition, the aspect context is encoded on both sides of the aspect target, so as to enhance the ability of the model to capture semantic information. The Multi-Attention mechanism (MATT) and Aspect Context (AC) allow our model to perform better when facing reviews with more complicated structures. The result of this experiment indicate that the accuracy of the new model is up to 80.6% and 75.1% for two datasets in SemEval-2014 Task 4 respectively, While the accuracy of the data set on twitter 71.1%, and 81.6% for the Chinese automotive-domain dataset. Compared with some previous models for sentiment analysis, our model shows a higher accuracy.

scholarly journals Online Measurement Error Detection for the ElectronicTransformer in a Smart Grid

Energies ◽  
2021 ◽  
Vol 14 (12) ◽  
pp. 3551
Author(s):  
Gu Xiong ◽  
Krzysztof Przystupa ◽  
Yao Teng ◽  
Wang Xue ◽  
Wang Huan ◽  
...  
Keyword(s):  
Error Detection ◽  
Field Experiments ◽  
Short Term Memory ◽  
Attention Mechanism ◽  
Power Grids ◽  
Monitoring Data ◽  
Smart Power ◽  
Long Term Stability ◽  
Smart Power Grids ◽  
Sequential Information

With the development of smart power grids, electronic transformers have been widely used to monitor the online status of power grids. However, electronic transformers have the drawback of poor long-term stability, leading to a requirement for frequent measurement. Aiming to monitor the online status frequently and conveniently, we proposed an attention mechanism-optimized Seq2Seq network to predict the error state of transformers, which combines an attention mechanism, Seq2Seq network, and bidirectional long short-term memory networks to mine the sequential information from online monitoring data of electronic transformers. We implemented the proposed method on the monitoring data of electronic transformers in a certain electric field. Experiments showed that our proposed attention mechanism-optimized Seq2Seq network has high accuracy in the aspect of error prediction.

Human Activity Recognition using Fourier Transform Inspired Deep Learning Combination Model

2019 ◽  
Vol 9 (1) ◽  
pp. 16-31
Author(s):  
Kyungkoo Jun
Keyword(s):  
Fourier Transform ◽  
Deep Learning ◽  
Short Term Memory ◽  
Window Size ◽  
Sensor Data ◽  
Data Sets ◽  
Data Set ◽  
Proposed Model ◽  
Testing Data ◽  
Labeling Scheme

Background & Objective: This paper proposes a Fourier transform inspired method to classify human activities from time series sensor data. Methods: Our method begins by decomposing 1D input signal into 2D patterns, which is motivated by the Fourier conversion. The decomposition is helped by Long Short-Term Memory (LSTM) which captures the temporal dependency from the signal and then produces encoded sequences. The sequences, once arranged into the 2D array, can represent the fingerprints of the signals. The benefit of such transformation is that we can exploit the recent advances of the deep learning models for the image classification such as Convolutional Neural Network (CNN). Results: The proposed model, as a result, is the combination of LSTM and CNN. We evaluate the model over two data sets. For the first data set, which is more standardized than the other, our model outperforms previous works or at least equal. In the case of the second data set, we devise the schemes to generate training and testing data by changing the parameters of the window size, the sliding size, and the labeling scheme. Conclusion: The evaluation results show that the accuracy is over 95% for some cases. We also analyze the effect of the parameters on the performance.

Sign in / Sign up

Export Citation Format

Share Document