scholarly journals Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification

2021 ◽  
Vol 11 (8) ◽  
pp. 3720
Author(s):  
Doyeob Yeo ◽  
Min-Suk Kim ◽  
Ji-Hoon Bae
Keyword(s):  
Knowledge Transfer ◽  
Euclidean Distance ◽  
Learning Technology ◽  
Transfer Scheme ◽  
Adversarial Network ◽  
Knowledge Distillation ◽  
Dense Flow ◽  
Adversarial Training ◽  
Transfer Method ◽  
Accuracy Performance

A deep-learning technology for knowledge transfer is necessary to advance and optimize efficient knowledge distillation. Here, we aim to develop a new adversarial optimization-based knowledge transfer method involved with a layer-wise dense flow that is distilled from a pre-trained deep neural network (DNN). Knowledge distillation transferred to another target DNN based on adversarial loss functions has multiple flow-based knowledge items that are densely extracted by overlapping them from a pre-trained DNN to enhance the existing knowledge. We propose a semi-supervised learning-based knowledge transfer with multiple items of dense flow-based knowledge extracted from the pre-trained DNN. The proposed loss function would comprise a supervised cross-entropy loss for a typical classification, an adversarial training loss for the target DNN and discriminators, and Euclidean distance-based loss in terms of dense flow. For both pre-trained and target DNNs considered in this study, we adopt a residual network (ResNet) architecture. We propose methods of (1) the adversarial-based knowledge optimization, (2) the extended and flow-based knowledge transfer scheme, and (3) the combined layer-wise dense flow in an adversarial network. The results show that it provides higher accuracy performance in the improved target ResNet compared to the prior knowledge transfer methods.

scholarly journals Hierarchical Knowledge Squeezed Adversarial Network Compression

2020 ◽  
Vol 34 (07) ◽  
pp. 11370-11377
Author(s):  
Peng Li ◽  
Chang Shu ◽  
Yuan Xie ◽  
Yan Qu ◽  
Hui Kong
Keyword(s):  
State Of The Art ◽  
Teacher Student ◽  
Adversarial Network ◽  
Benchmark Datasets ◽  
Knowledge Distillation ◽  
Adversarial Training ◽  
Rich Information ◽  
Process Oriented ◽  
Transfer Method ◽  
Network Compression

Deep network compression has been achieved notable progress via knowledge distillation, where a teacher-student learning manner is adopted by using predetermined loss. Recently, more focuses have been transferred to employ the adversarial training to minimize the discrepancy between distributions of output from two networks. However, they always emphasize on result-oriented learning while neglecting the scheme of process-oriented learning, leading to the loss of rich information contained in the whole network pipeline. Whereas in other (non GAN-based) process-oriented methods, the knowledge have usually been transferred in a redundant manner. Observing that, the small network can not perfectly mimic a large one due to the huge gap of network scale, we propose a knowledge transfer method, involving effective intermediate supervision, under the adversarial training framework to learn the student network. Different from the other intermediate supervision methods, we design the knowledge representation in a compact form by introducing a task-driven attention mechanism. Meanwhile, to improve the representation capability of the attention-based method, a hierarchical structure is utilized so that powerful but highly squeezed knowledge is realized and the knowledge from teacher network could accommodate the size of student network. Extensive experimental results on three typical benchmark datasets, i.e., CIFAR-10, CIFAR-100, and ImageNet, demonstrate that our method achieves highly superior performances against state-of-the-art methods.

Role of General Adversarial Networks in Mammogram Analysis: A Review

2020 ◽  
Vol 16 (7) ◽  
pp. 863-877
Author(s):  
Annapoorani Gopal ◽  
Lathaselvi Gandhimaruthian ◽  
Javid Ali
Keyword(s):  
Breast Tumor ◽  
Deep Neural Networks ◽  
Training Data ◽  
Learning Technology ◽  
Breast Cancers ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Adversarial Networks ◽  
Tumor Extraction

The Deep Neural Networks have gained prominence in the biomedical domain, becoming the most commonly used networks after machine learning technology. Mammograms can be used to detect breast cancers with high precision with the help of Convolutional Neural Network (CNN) which is deep learning technology. An exhaustive labeled data is required to train the CNN from scratch. This can be overcome by deploying Generative Adversarial Network (GAN) which comparatively needs lesser training data during a mammogram screening. In the proposed study, the application of GANs in estimating breast density, high-resolution mammogram synthesis for clustered microcalcification analysis, effective segmentation of breast tumor, analysis of the shape of breast tumor, extraction of features and augmentation of the image during mammogram classification have been extensively reviewed.

Book Works’ Archive: a partial response

2013 ◽  
Vol 38 (3) ◽  
pp. 36-41
Author(s):  
Karen Di Franco
Keyword(s):  
Knowledge Transfer ◽  
Partial Response ◽  
Case Studies ◽  
Classification System ◽  
Research Centre ◽  
Transfer Scheme ◽  
The Arts ◽  
Select Material ◽  
Search Facility ◽  
Working Processes

Since 2010, Book Works has been digitising material from its archive – whether finished works, ephemera, correspondence, photographs, or manuscripts – to give access to the working processes of the organisation (at www.bookworks.org.uk). The archive database is constructed around a chronological timeline and includes a search facility that allows visitors to filter and select material using a bespoke classification system. It currently comprises detailed content relating to two case studies from Book Works back catalogue: After the Freud Museum by Susan Hiller and Erasmus is late by Liam Gillick, as well as ephemera and material from other works. The project has been developed in collaboration with Ligatus Research Centre, University of the Arts London, with support from the AHRC Knowledge Transfer scheme.

scholarly journals Robust CNN Compression Framework for Security-Sensitive Embedded Systems

2021 ◽  
Vol 11 (3) ◽  
pp. 1093
Author(s):  
Jeonghyun Lee ◽  
Sangkyun Lee
Keyword(s):  
Embedded Systems ◽  
Optimization Problem ◽  
State Of The Art ◽  
Classification Problems ◽  
Proximal Gradient Method ◽  
Knowledge Distillation ◽  
New Type ◽  
Adversarial Examples ◽  
Adversarial Training ◽  
Memory Efficient

Convolutional neural networks (CNNs) have achieved tremendous success in solving complex classification problems. Motivated by this success, there have been proposed various compression methods for downsizing the CNNs to deploy them on resource-constrained embedded systems. However, a new type of vulnerability of compressed CNNs known as the adversarial examples has been discovered recently, which is critical for security-sensitive systems because the adversarial examples can cause malfunction of CNNs and can be crafted easily in many cases. In this paper, we proposed a compression framework to produce compressed CNNs robust against such adversarial examples. To achieve the goal, our framework uses both pruning and knowledge distillation with adversarial training. We formulate our framework as an optimization problem and provide a solution algorithm based on the proximal gradient method, which is more memory-efficient than the popular ADMM-based compression approaches. In experiments, we show that our framework can improve the trade-off between adversarial robustness and compression rate compared to the existing state-of-the-art adversarial pruning approach.

scholarly journals A stacked dense denoising–segmentation network for undersampled tomograms and knowledge transfer using synthetic tomograms

2021 ◽  
Vol 32 (3) ◽  
Author(s):  
Dimitrios Bellos ◽  
Mark Basham ◽  
Tony Pridmore ◽  
Andrew P. French
Keyword(s):  
Time Series ◽  
Knowledge Transfer ◽  
Semantic Segmentation ◽  
Real World Data ◽  
Transfer Scheme ◽  
X Ray ◽  
Segmentation Accuracy ◽  
X Ray Computed ◽  
Temporal Events ◽  
Time Critical

AbstractOver recent years, many approaches have been proposed for the denoising or semantic segmentation of X-ray computed tomography (CT) scans. In most cases, high-quality CT reconstructions are used; however, such reconstructions are not always available. When the X-ray exposure time has to be limited, undersampled tomograms (in terms of their component projections) are attained. This low number of projections offers low-quality reconstructions that are difficult to segment. Here, we consider CT time-series (i.e. 4D data), where the limited time for capturing fast-occurring temporal events results in the time-series tomograms being necessarily undersampled. Fortunately, in these collections, it is common practice to obtain representative highly sampled tomograms before or after the time-critical portion of the experiment. In this paper, we propose an end-to-end network that can learn to denoise and segment the time-series’ undersampled CTs, by training with the earlier highly sampled representative CTs. Our single network can offer two desired outputs while only training once, with the denoised output improving the accuracy of the final segmentation. Our method is able to outperform state-of-the-art methods in the task of semantic segmentation and offer comparable results in regard to denoising. Additionally, we propose a knowledge transfer scheme using synthetic tomograms. This not only allows accurate segmentation and denoising using less real-world data, but also increases segmentation accuracy. Finally, we make our datasets, as well as the code, publicly available.

scholarly journals A Confrontation Decision-Making Method with Deep Reinforcement Learning and Knowledge Transfer for Multi-Agent System

Symmetry ◽  
2020 ◽  
Vol 12 (4) ◽  
pp. 631
Author(s):  
Chunyang Hu
Keyword(s):  
Decision Making ◽  
Reinforcement Learning ◽  
Knowledge Transfer ◽  
Large Scale ◽  
Effective Control ◽  
Small Scale ◽  
Learning Agent ◽  
Multi Agent ◽  
Transfer Method ◽  
Parameter Sharing

In this paper, deep reinforcement learning (DRL) and knowledge transfer are used to achieve the effective control of the learning agent for the confrontation in the multi-agent systems. Firstly, a multi-agent Deep Deterministic Policy Gradient (DDPG) algorithm with parameter sharing is proposed to achieve confrontation decision-making of multi-agent. In the process of training, the information of other agents is introduced to the critic network to improve the strategy of confrontation. The parameter sharing mechanism can reduce the loss of experience storage. In the DDPG algorithm, we use four neural networks to generate real-time action and Q-value function respectively and use a momentum mechanism to optimize the training process to accelerate the convergence rate for the neural network. Secondly, this paper introduces an auxiliary controller using a policy-based reinforcement learning (RL) method to achieve the assistant decision-making for the game agent. In addition, an effective reward function is used to help agents balance losses of enemies and our side. Furthermore, this paper also uses the knowledge transfer method to extend the learning model to more complex scenes and improve the generalization of the proposed confrontation model. Two confrontation decision-making experiments are designed to verify the effectiveness of the proposed method. In a small-scale task scenario, the trained agent can successfully learn to fight with the competitors and achieve a good winning rate. For large-scale confrontation scenarios, the knowledge transfer method can gradually improve the decision-making level of the learning agent.

scholarly journals The Missing Data Encoder: Cross-Channel Image Completion with Hide-and-Seek Adversarial Network

2020 ◽  
Vol 34 (07) ◽  
pp. 10688-10695
Author(s):  
Arnaud Dapogny ◽  
Matthieu Cord ◽  
Patrick Perez
Keyword(s):  
Missing Data ◽  
Representation Learning ◽  
Image Completion ◽  
Occlusion Handling ◽  
Deep Network ◽  
Adversarial Network ◽  
Image Semantics ◽  
Adversarial Training ◽  
Channel Image

Image completion is the problem of generating whole images from fragments only. It encompasses inpainting (generating a patch given its surrounding), reverse inpainting/extrapolation (generating the periphery given the central patch) as well as colorization (generating one or several channels given other ones). In this paper, we employ a deep network to perform image completion, with adversarial training as well as perceptual and completion losses, and call it the “missing data encoder” (MDE). We consider several configurations based on how the seed fragments are chosen. We show that training MDE for “random extrapolation and colorization” (MDE-REC), i.e. using random channel-independent fragments, allows a better capture of the image semantics and geometry. MDE training makes use of a novel “hide-and-seek” adversarial loss, where the discriminator seeks the original non-masked regions, while the generator tries to hide them. We validate our models qualitatively and quantitatively on several datasets, showing their interest for image completion, representation learning as well as face occlusion handling.

scholarly journals An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension

2020 ◽  
Author(s):  
Xin Liu ◽  
Kai Liu ◽  
Xiang Li ◽  
Jinsong Su ◽  
Yubin Ge ◽  
...  
Keyword(s):  
Reading Comprehension ◽  
Knowledge Transfer ◽  
Training Data ◽  
Target Domain ◽  
Domain Specific ◽  
Mutual Knowledge ◽  
Benchmark Datasets ◽  
Knowledge Distillation ◽  
The Many ◽  
Machine Reading

The lack of sufficient training data in many domains, poses a major challenge to the construction of domain-specific machine reading comprehension (MRC) models with satisfying performance. In this paper, we propose a novel iterative multi-source mutual knowledge transfer framework for MRC. As an extension of the conventional knowledge transfer with one-to-one correspondence, our framework focuses on the many-to-many mutual transfer, which involves synchronous executions of multiple many-to-one transfers in an iterative manner.Specifically, to update a target-domain MRC model, we first consider other domain-specific MRC models as individual teachers, and employ knowledge distillation to train a multi-domain MRC model, which is differentially required to fit the training data and match the outputs of these individual models according to their domain-level similarities to the target domain. After being initialized by the multi-domain MRC model, the target-domain MRC model is fine-tuned to match both its training data and the output of its previous best model simultaneously via knowledge distillation. Compared with previous approaches, our framework can continuously enhance all domain-specific MRC models by enabling each model to iteratively and differentially absorb the domain-shared knowledge from others. Experimental results and in-depth analyses on several benchmark datasets demonstrate the effectiveness of our framework.

Sign in / Sign up

Export Citation Format

Share Document