Optimization methods for deep neural networks

2021 ◽  
Author(s):  
Rajendra P. ◽  
Hanumantha Ravi. P. V. N. ◽  
Gunavardhana Naidu T.
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Optimization Methods

scholarly journals Overparameterized neural networks implement associative memory

2020 ◽  
Vol 117 (44) ◽  
pp. 27162-27170
Author(s):  
Adityanarayanan Radhakrishnan ◽  
Mikhail Belkin ◽  
Caroline Uhler
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Associative Memory ◽  
Deep Neural Networks ◽  
Optimization Methods ◽  
Efficient Mechanism ◽  
Training Samples ◽  
Sample Recovery ◽  
Sequence Encoding ◽  
Standing Problem

Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. We provide empirical evidence that 1) overparameterized autoencoders store training samples as attractors and thus iterating the learned map leads to sample recovery, and that 2) the same mechanism allows for encoding sequences of examples and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.

scholarly journals Inverse Design for Silicon Photonics: From Iterative Optimization Algorithms to Deep Neural Networks

2021 ◽  
Vol 11 (9) ◽  
pp. 3822
Author(s):  
Simei Mao ◽  
Lirong Cheng ◽  
Caiyue Zhao ◽  
Faisal Nadeem Khan ◽  
Qian Li ◽  
...  
Keyword(s):  
Neural Networks ◽  
Silicon Photonics ◽  
Design Methodology ◽  
Degrees Of Freedom ◽  
Deep Neural Networks ◽  
Low Cost ◽  
Optimization Algorithms ◽  
Optimization Methods ◽  
Inverse Design ◽  
Iterative Optimization

Silicon photonics is a low-cost and versatile platform for various applications. For design of silicon photonic devices, the light-material interaction within its complex subwavelength geometry is difficult to investigate analytically and therefore numerical simulations are majorly adopted. To make the design process more time-efficient and to improve the device performance to its physical limits, various methods have been proposed over the past few years to manipulate the geometries of silicon platform for specific applications. In this review paper, we summarize the design methodologies for silicon photonics including iterative optimization algorithms and deep neural networks. In case of iterative optimization methods, we discuss them in different scenarios in the sequence of increased degrees of freedom: empirical structure, QR-code like structure and irregular structure. We also review inverse design approaches assisted by deep neural networks, which generate multiple devices with similar structure much faster than iterative optimization methods and are thus suitable in situations where piles of optical components are needed. Finally, the applications of inverse design methodology in optical neural networks are also discussed. This review intends to provide the readers with the suggestion for the most suitable design methodology for a specific scenario.

Deep neural networks trained with heavier data augmentation learn features closer to representations in hIT

2018 ◽  
Author(s):  
Alex Hernández-García ◽  
Johannes Mehrer ◽  
Nikolaus Kriegeskorte ◽  
Peter König ◽  
Tim C. Kietzmann
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Data Augmentation

Representation of adversarial images in deep neural networks and the human brain

2018 ◽  
Author(s):  
Chi Zhang ◽  
Xiaohan Duan ◽  
Ruyuan Zhang ◽  
Li Tong
Keyword(s):  
Neural Networks ◽  
Human Brain ◽  
Deep Neural Networks
Sign in / Sign up

Export Citation Format

Share Document