scholarly journals The Old and the New: Can Physics-Informed Deep-Learning Replace Traditional Linear Solvers?

2021 ◽  
Vol 4 ◽  
Author(s):  
Stefano Markidis
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
High Performance ◽  
Critical Role ◽  
Low Frequency ◽  
Accurate Solution ◽  
Limiting Factor ◽  
Data Set ◽  
Linear Solvers ◽  
Computational Performance

Physics-Informed Neural Networks (PINN) are neural networks encoding the problem governing equations, such as Partial Differential Equations (PDE), as a part of the neural network. PINNs have emerged as a new essential tool to solve various challenging problems, including computing linear systems arising from PDEs, a task for which several traditional methods exist. In this work, we focus first on evaluating the potential of PINNs as linear solvers in the case of the Poisson equation, an omnipresent equation in scientific computing. We characterize PINN linear solvers in terms of accuracy and performance under different network configurations (depth, activation functions, input data set distribution). We highlight the critical role of transfer learning. Our results show that low-frequency components of the solution converge quickly as an effect of the F-principle. In contrast, an accurate solution of the high frequencies requires an exceedingly long time. To address this limitation, we propose integrating PINNs into traditional linear solvers. We show that this integration leads to the development of new solvers whose performance is on par with other high-performance solvers, such as PETSc conjugate gradient linear solvers, in terms of performance and accuracy. Overall, while the accuracy and computational performance are still a limiting factor for the direct use of PINN linear solvers, hybrid strategies combining old traditional linear solver approaches with new emerging deep-learning techniques are among the most promising methods for developing a new class of linear solvers.

scholarly journals Aspects of programming for implementation of convolutional neural networks on multisystem HPC architectures

2021 ◽  
Vol 2062 (1) ◽  
pp. 012016
Author(s):  
Sunil Pandey ◽  
Naresh Kumar Nagwani ◽  
Shrish Verma
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
High Performance Computing ◽  
Convolutional Neural Networks ◽  
High Performance ◽  
Processing Unit ◽  
Computational Performance ◽  
Performance Computing

Abstract The training of deep learning convolutional neural networks is extremely compute intensive and takes long times for completion, on all except small datasets. This is a major limitation inhibiting the widespread adoption of convolutional neural networks in real world applications despite their better image classification performance in comparison with other techniques. Multidirectional research and development efforts are therefore being pursued with the objective of boosting the computational performance of convolutional neural networks. Development of parallel and scalable deep learning convolutional neural network implementations for multisystem high performance computing architectures is important in this background. Prior analysis based on computational experiments indicates that a combination of pipeline and task parallelism results in significant convolutional neural network performance gains of up to 18 times. This paper discusses the aspects which are important from the perspective of implementation of parallel and scalable convolutional neural networks on central processing unit based multisystem high performance computing architectures including computational pipelines, convolutional neural networks, convolutional neural network pipelines, multisystem high performance computing architectures and parallel programming models.

scholarly journals Computational Complexity Reduction of Neural Networks of Brain Tumor Image Segmentation by Introducing Fermi–Dirac Correction Functions

Entropy ◽  
2021 ◽  
Vol 23 (2) ◽  
pp. 223
Author(s):  
Yen-Ling Tai ◽  
Shin-Jhe Huang ◽  
Chien-Chang Chen ◽  
Henry Horng-Shing Lu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Computational Complexity ◽  
High Performance ◽  
Low Cost ◽  
Structural Complexity ◽  
Correction Function ◽  
Computational Time ◽  
Learning Methods ◽  
Band Theory

Nowadays, deep learning methods with high structural complexity and flexibility inevitably lean on the computational capability of the hardware. A platform with high-performance GPUs and large amounts of memory could support neural networks having large numbers of layers and kernels. However, naively pursuing high-cost hardware would probably drag the technical development of deep learning methods. In the article, we thus establish a new preprocessing method to reduce the computational complexity of the neural networks. Inspired by the band theory of solids in physics, we map the image space into a noninteraction physical system isomorphically and then treat image voxels as particle-like clusters. Then, we reconstruct the Fermi–Dirac distribution to be a correction function for the normalization of the voxel intensity and as a filter of insignificant cluster components. The filtered clusters at the circumstance can delineate the morphological heterogeneity of the image voxels. We used the BraTS 2019 datasets and the dimensional fusion U-net for the algorithmic validation, and the proposed Fermi–Dirac correction function exhibited comparable performance to other employed preprocessing methods. By comparing to the conventional z-score normalization function and the Gamma correction function, the proposed algorithm can save at least 38% of computational time cost under a low-cost hardware architecture. Even though the correction function of global histogram equalization has the lowest computational time among the employed correction functions, the proposed Fermi–Dirac correction function exhibits better capabilities of image augmentation and segmentation.

Wavelets as activation functions in Neural Networks

2021 ◽  
pp. 1-11
Author(s):  
Oscar Herrera ◽  
Belém Priego
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
High Performance ◽  
Research Area ◽  
Experimental Results ◽  
Activation Functions ◽  
Hyperbolic Tangent ◽  
Open Research ◽  
Bounded Functions

Traditionally, a few activation functions have been considered in neural networks, including bounded functions such as threshold, sigmoidal and hyperbolic-tangent, as well as unbounded ReLU, GELU, and Soft-plus, among other functions for deep learning, but the search for new activation functions still being an open research area. In this paper, wavelets are reconsidered as activation functions in neural networks and the performance of Gaussian family wavelets (first, second and third derivatives) are studied together with other functions available in Keras-Tensorflow. Experimental results show how the combination of these activation functions can improve the performance and supports the idea of extending the list of activation functions to wavelets which can be available in high performance platforms.

scholarly journals Neural network activation similarity: a new measure to assist decision making in chemical toxicology

2020 ◽  
Vol 11 (28) ◽  
pp. 7335-7348 ◽  
Author(s):  
Timothy E. H. Allen ◽  
Andrew J. Wedlake ◽  
Elena Gelžinytė ◽  
Charles Gong ◽  
Jonathan M. Goodman ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Decision Making ◽  
Deep Learning ◽  
Test Data ◽  
High Performance ◽  
Chemical Binding ◽  
Biological Targets ◽  
Network Activation ◽  
Chemical Toxicology

Deep learning neural networks, constructed for the prediction of chemical binding at 79 pharmacologically important human biological targets, show extremely high performance on test data (accuracy 92.2 ± 4.2%, MCC 0.814 ± 0.093, ROC-AUC 0.96 ± 0.04).

scholarly journals Optimizing Convolution Neural Network on the TI C6678 multicore DSP

2018 ◽  
Vol 246 ◽  
pp. 03044 ◽  
Author(s):  
Guozhao Zeng ◽  
Xiao Hu ◽  
Yueyue Chen
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Image Processing ◽  
Deep Learning ◽  
Digital Signal Processor ◽  
High Performance ◽  
Digital Signal ◽  
Automatic Translation ◽  
Convolution Operation ◽  
Multicore Dsp

Convolutional Neural Networks (CNNs) have become the most advanced algorithms for deep learning. They are widely used in image processing, object detection and automatic translation. As the demand for CNNs continues to increase, the platforms on which they are deployed continue to expand. As an excellent low-power, high-performance, embedded solution, Digital Signal Processor (DSP) is used frequently in many key areas. This paper attempts to deploy the CNN to Texas Instruments (TI)’s TMS320C6678 multi-core DSP and optimize the main operations (convolution) to accommodate the DSP structure. The efficiency of the improved convolution operation has increased by tens of times.

scholarly journals A New Deep Learning Pipeline to Detect Covid-19 on Chest X-Ray Images using Local Binary Pattern, Dual Tree Complex Wavelet Transform and Convolutional Neural Networks

2020 ◽  
Author(s):  
Huseyin Yaşar ◽  
Murat Ceylan
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Wavelet Transform ◽  
Test Data ◽  
Local Binary Pattern ◽  
Complex Wavelet Transform ◽  
Data Set ◽  
X Ray ◽  
Chest X Ray ◽  
Sensitivity Specificity

Abstract At the end of 2019, a new type of virus, belonging to the coronaviridae family has emerged and it is considered that the virus in question is of zootonic origin. The virus that emerged in China first affected this country and then spread worldwide. Pneumonia develops due to Covid-19 virus in patients having severe disease symptoms. Many literature studies have been carried out in the process where the effects of the disease-induced pneumonia in lungs have been demonstrated with the help of chest X-ray imaging. In this study, which aims at early diagnosis of Covid-19 disease by using X-Ray images, the deep-learning approach, which is a state-of-the-art artificial intelligence method, was used and automatic classification of images was performed using Convolutional Neural Networks (CNN). In the first training-test data set used in the study, there were a total of 230 abnormal and 80 normal X-Ray images, while in the second training-test data set there were 476 X-Ray images, of which 150 abnormal and 326 normal. Thus, classification results have been provided for two data sets, containing predominantly abnormal images and predominantly normal images respectively. In the study, a 23-layer CNN architecture was developed. Within the scope of the study, results were obtained by using chest X-Ray images directly in training-test procedures and the sub-band images obtained by applying Dual Tree Complex Wavelet Transform (DT-CWT) to the above-mentioned images. The same experiments were repeated using images obtained by applying Local Binary Pattern (LBP) to the chest X-Ray images. Within the scope of the study, a new result generation algorithm having been put forward additionally, it was ensured that the experimental results were combined and the success of the study was improved. In the experiments carried out in the study, the trainings were carried out using the k-fold cross validation method. Here the k value was chosen 23. Considering the highest results of the tests performed in the study, values of sensitivity, specificity, accuracy and AUC for the first training-test data set were calculated to be 1, 1, 0,9913 and 0,9996; while for the second data set of training-test, they were 1, 0,9969, 0,9958 and 0,9996 respectively. Considering the average highest results of the experiments performed within the scope of the study, the values of sensitivity, specificity, accuracy and AUC for the first training-test data set were 0,9933, 0,9725, 0,9843 and 0,9988; while for the second training-test data set, they were 0,9813, 0,9908, 0,9857 and 0,9983 respectively.

scholarly journals A convolutional neural network for predicting transcriptional regulators of genes in Arabidopsis transcriptome data reveals classification based on positive regulatory interactions

2019 ◽  
Author(s):  
Dan MacLean
Keyword(s):  
Neural Network ◽  
Gene Expression ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Expression Profiles ◽  
Biological Data ◽  
Expression Data ◽  
Data Set

AbstractGene Regulatory networks that control gene expression are widely studied yet the interactions that make them up are difficult to predict from high throughput data. Deep Learning methods such as convolutional neural networks can perform surprisingly good classifications on a variety of data types and the matrix-like gene expression profiles would seem to be ideal input data for deep learning approaches. In this short study I compiled training sets of expression data using the Arabidopsis AtGenExpress global stress expression data set and known transcription factor-target interactions from the Arabidopsis PLACE database. I built and optimised convolutional neural networks with a best model providing 95 % accuracy of classification on a held-out validation set. Investigation of the activations within this model revealed that classification was based on positive correlation of expression profiles in short sections. This result shows that a convolutional neural network can be used to make classifications and reveal the basis of those calssifications for gene expression data sets, indicating that a convolutional neural network is a useful and interpretable tool for exploratory classification of biological data. The final model is available for download and as a web application.

scholarly journals A review of the use of convolutional neural networks in agriculture

2018 ◽  
Vol 156 (3) ◽  
pp. 312-322 ◽  
Author(s):  
A. Kamilaris ◽  
F. X. Prenafeta-Boldú
Keyword(s):  
Neural Networks ◽  
Image Processing ◽  
Convolutional Neural Networks ◽  
High Performance ◽  
Performance Metrics ◽  
Specific Class ◽  
Data Set ◽  
Advantages And Disadvantages ◽  
Future Potential ◽  
Processing Techniques

AbstractDeep learning (DL) constitutes a modern technique for image processing, with large potential. Having been successfully applied in various areas, it has recently also entered the domain of agriculture. In the current paper, a survey was conducted of research efforts that employ convolutional neural networks (CNN), which constitute a specific class of DL, applied to various agricultural and food production challenges. The paper examines agricultural problems under study, models employed, sources of data used and the overall precision achieved according to the performance metrics used by the authors. Convolutional neural networks are compared with other existing techniques, and the advantages and disadvantages of using CNN in agriculture are listed. Moreover, the future potential of this technique is discussed, together with the authors’ personal experiences after employing CNN to approximate a problem of identifying missing vegetation from a sugar cane plantation in Costa Rica. The overall findings indicate that CNN constitutes a promising technique with high performance in terms of precision and classification accuracy, outperforming existing commonly used image-processing techniques. However, the success of each CNN model is highly dependent on the quality of the data set used.

scholarly journals AN APPROACH TO ACCELERATE THE TRAINING OF CONVOLUTIONAL NEURAL NETWORKS BY TUNING THE HYPERPARAMETERS OF LEARNING

2020 ◽  
Vol 2 (2) ◽  
pp. 32-37
Author(s):  
P. RADIUK ◽  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Video Processing ◽  
Network Architecture ◽  
Random Search ◽  
Machine Learning Algorithms ◽  
Language Recognition ◽  
Data Set ◽  
Training Time

Over the last decade, a set of machine learning algorithms called deep learning has led to significant improvements in computer vision, natural language recognition and processing. This has led to the widespread use of a variety of commercial, learning-based products in various fields of human activity. Despite this success, the use of deep neural networks remains a black box. Today, the process of setting hyperparameters and designing a network architecture requires experience and a lot of trial and error and is based more on chance than on a scientific approach. At the same time, the task of simplifying deep learning is extremely urgent. To date, no simple ways have been invented to establish the optimal values of learning hyperparameters, namely learning speed, sample size, data set, learning pulse, and weight loss. Grid search and random search of hyperparameter space are extremely resource intensive. The choice of hyperparameters is critical for the training time and the final result. In addition, experts often choose one of the standard architectures (for example, ResNets and ready-made sets of hyperparameters. However, such kits are usually suboptimal for specific practical tasks. The presented work offers an approach to finding the optimal set of hyperparameters of learning ZNM. An integrated approach to all hyperparameters is valuable because there is an interdependence between them. The aim of the work is to develop an approach for setting a set of hyperparameters, which will reduce the time spent during the design of ZNM and ensure the efficiency of its work. In recent decades, the introduction of deep learning methods, in particular convolutional neural networks (CNNs), has led to impressive success in image and video processing. However, the training of CNN has been commonly mostly based on the employment of quasi-optimal hyperparameters. Such an approach usually requires huge computational and time costs to train the network and does not guarantee a satisfactory result. However, hyperparameters play a crucial role in the effectiveness of CNN, as diverse hyperparameters lead to models with significantly different characteristics. Poorly selected hyperparameters generally lead to low model performance. The issue of choosing optimal hyperparameters for CNN has not been resolved yet. The presented work proposes several practical approaches to setting hyperparameters, which allows reducing training time and increasing the accuracy of the model. The article considers the function of training validation loss during underfitting and overfitting. There are guidelines in the end to reach the optimization point. The paper also considers the regulation of learning rate and momentum to accelerate network training. All experiments are based on the widespread CIFAR-10 and CIFAR-100 datasets.

Sign in / Sign up

Export Citation Format

Share Document