scholarly journals Seg2pix: Few Shot Training Line Art Colorization with Segmented Image Data

2021 ◽  
Vol 11 (4) ◽  
pp. 1464
Author(s):  
Chang Wook Seo ◽  
Yongduek Seo
Keyword(s):  
Image Data ◽  
Ground Truth ◽  
Semantic Segmentation ◽  
Training Data ◽  
High Quality ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Segmented Image ◽  
Image Translation ◽  
Segmentation Image

There are various challenging issues in automating line art colorization. In this paper, we propose a GAN approach incorporating semantic segmentation image data. Our GAN-based method, named Seg2pix, can automatically generate high quality colorized images, aiming at computerizing one of the most tedious and repetitive jobs performed by coloring workers in the webtoon industry. The network structure of Seg2pix is mostly a modification of the architecture of Pix2pix, which is a convolution-based generative adversarial network for image-to-image translation. Through this method, we can generate high quality colorized images of a particular character with only a few training data. Seg2pix is designed to reproduce a segmented image, which becomes the suggestion data for line art colorization. The segmented image is automatically generated through a generative network with a line art image and a segmentation ground truth. In the next step, this generative network creates a colorized image from the line art and segmented image, which is generated from the former step of the generative network. To summarize, only one line art image is required for testing the generative model, and an original colorized image and segmented image are additionally required as the ground truth for training the model. These generations of the segmented image and colorized image proceed by an end-to-end method sharing the same loss functions. By using this method, we produce better qualitative results for automatic colorization of a particular character’s line art. This improvement can also be measured by quantitative results with Learned Perceptual Image Patch Similarity (LPIPS) comparison. We believe this may help artists exercise their creative expertise mainly in the area where computerization is not yet capable.

scholarly journals Synthesizing Depth Hand Images with GANs and Style Transfer for Hand Pose Estimation

Sensors ◽  
2019 ◽  
Vol 19 (13) ◽  
pp. 2919 ◽  
Author(s):  
Wangyong He ◽  
Zhongzhao Xie ◽  
Yongbo Li ◽  
Xinmei Wang ◽  
Wendi Cai
Keyword(s):  
Pose Estimation ◽  
Ground Truth ◽  
Training Image ◽  
Training Data ◽  
Generative Adversarial Network ◽  
Style Transfer ◽  
Visual Appearance ◽  
Hand Pose Estimation ◽  
Adversarial Network ◽  
Hand Pose

Hand pose estimation is a critical technology of computer vision and human-computer interaction. Deep-learning methods require a considerable amount of tagged data. Accordingly, numerous labeled training data are required. This paper aims to generate depth hand images. Given a ground-truth 3D hand pose, the developed method can generate depth hand images. To be specific, a ground truth can be 3D hand poses with the hand structure contained, while the synthesized image has an identical size to that of the training image and a similar visual appearance to the training set. The developed method, inspired by the progress in the generative adversarial network (GAN) and image-style transfer, helps model the latent statistical relationship between the ground-truth hand pose and the corresponding depth hand image. The images synthesized using the developed method are demonstrated to be feasible for enhancing performance. On public hand pose datasets (NYU, MSRA, ICVL), comprehensive experiments prove that the developed method outperforms the existing works.

PaDGAN: Learning to Generate High-Quality Novel Designs

2020 ◽  
Vol 143 (3) ◽  
Author(s):  
Wei Chen ◽  
Faez Ahmed
Keyword(s):  
Loss Function ◽  
Design Space Exploration ◽  
Design Space ◽  
Space Exploration ◽  
Generative Models ◽  
Training Data ◽  
High Quality ◽  
Generative Adversarial Network ◽  
Design Synthesis ◽  
Adversarial Network

Abstract Deep generative models are proven to be a useful tool for automatic design synthesis and design space exploration. When applied in engineering design, existing generative models face three challenges: (1) generated designs lack diversity and do not cover all areas of the design space, (2) it is difficult to explicitly improve the overall performance or quality of generated designs, and (3) existing models generally do not generate novel designs, outside the domain of the training data. In this article, we simultaneously address these challenges by proposing a new determinantal point process-based loss function for probabilistic modeling of diversity and quality. With this new loss function, we develop a variant of the generative adversarial network, named “performance augmented diverse generative adversarial network” (PaDGAN), which can generate novel high-quality designs with good coverage of the design space. By using three synthetic examples and one real-world airfoil design example, we demonstrate that PaDGAN can generate diverse and high-quality designs. In comparison to a vanilla generative adversarial network, on average, it generates samples with a 28% higher mean quality score with larger diversity and without the mode collapse issue. Unlike typical generative models that usually generate new designs by interpolating within the boundary of training data, we show that PaDGAN expands the design space boundary outside the training data towards high-quality regions. The proposed method is broadly applicable to many tasks including design space exploration, design optimization, and creative solution recommendation.

scholarly journals An Adversarial and Densely Dilated Network for Connectomes Segmentation

Symmetry ◽  
2018 ◽  
Vol 10 (10) ◽  
pp. 467 ◽  
Author(s):  
Ke Chen ◽  
Dandan Zhu ◽  
Jianwei Lu ◽  
Ye Luo
Keyword(s):  
State Of The Art ◽  
Piriform Cortex ◽  
Receptive Fields ◽  
Ground Truth ◽  
Training Data ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Boundary Map ◽  
Segmentation Task ◽  
The Brain

Automatic reconstructing of neural circuits in the brain is one of the most crucial studies in neuroscience. Connectomes segmentation plays an important role in reconstruction from electron microscopy (EM) images; however, it is rather challenging due to highly anisotropic shapes with inferior quality and various thickness. In our paper, we propose a novel connectomes segmentation framework called adversarial and densely dilated network (ADDN) to address these issues. ADDN is based on the conditional Generative Adversarial Network (cGAN) structure which is the latest advance in machine learning with power to generate images similar to the ground truth especially when the training data is limited. Specifically, we design densely dilated network (DDN) as the segmentor to allow a deeper architecture and larger receptive fields for more accurate segmentation. Discriminator is trained to distinguish generated segmentation from manual segmentation. During training, such adversarial loss function is optimized together with dice loss. Extensive experimental results demonstrate that our ADDN is effective for such connectomes segmentation task, helping to retrieve more accurate segmentation and attenuate the blurry effects of generated boundary map. Our method obtains state-of-the-art performance while requiring less computation on ISBI 2012 EM dataset and mouse piriform cortex dataset.

scholarly journals Proactive Construction of an Annotated Imaging Database for Artificial Intelligence Training

2020 ◽  
Author(s):  
Caroline Bivik Stadler ◽  
Martin Lindvall ◽  
Claes Lundström ◽  
Anna Bodén ◽  
Karin Lindman ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Image Data ◽  
Ground Truth ◽  
Lessons Learned ◽  
Training Data ◽  
Imaging Data ◽  
High Quality ◽  
High Quality Image ◽  
Specific Objective ◽  
Wide Range

Abstract Artificial intelligence (AI) holds much promise for enabling highly desired imaging diagnostics improvements. One of the most limiting bottlenecks for the development of useful clinical-grade AI models is the lack of training data. One aspect is the large amount of cases needed and another is the necessity of high-quality ground truth annotation. The aim of the project was to establish and describe the construction of a database with substantial amounts of detail-annotated oncology imaging data from pathology and radiology. A specific objective was to be proactive, that is, to support undefined subsequent AI training across a wide range of tasks, such as detection, quantification, segmentation, and classification, which puts particular focus on the quality and generality of the annotations. The main outcome of this project was the database as such, with a collection of labeled image data from breast, ovary, skin, colon, skeleton, and liver. In addition, this effort also served as an exploration of best practices for further scalability of high-quality image collections, and a main contribution of the study was generic lessons learned regarding how to successfully organize efforts to construct medical imaging databases for AI training, summarized as eight guiding principles covering team, process, and execution aspects.

scholarly journals PaDGAN: A Generative Adversarial Network for Performance Augmented Diverse Designs

2020 ◽  
Author(s):  
Wei Chen ◽  
Faez Ahmed
Keyword(s):  
Loss Function ◽  
Design Space Exploration ◽  
Point Processes ◽  
Design Space ◽  
Space Exploration ◽  
Generative Models ◽  
Training Data ◽  
High Quality ◽  
Generative Adversarial Network ◽  
Adversarial Network

Abstract Deep generative models are proven to be a useful tool for automatic design synthesis and design space exploration. When applied in engineering design, existing generative models face three challenges: 1) generated designs lack diversity and do not cover all areas of the design space, 2) it is difficult to explicitly improve the overall performance or quality of generated designs, and 3) existing models generate do not generate novel designs, outside the domain of the training data. In this paper, we simultaneously address these challenges by proposing a new Determinantal Point Processes based loss function for probabilistic modeling of diversity and quality. With this new loss function, we develop a variant of the Generative Adversarial Network, named “Performance Augmented Diverse Generative Adversarial Network” or PaDGAN, which can generate novel high-quality designs with good coverage of the design space. Using three synthetic examples and one real-world airfoil design example, we demonstrate that PaDGAN can generate diverse and high-quality designs. In comparison to a vanilla Generative Adversarial Network, on average, it generates samples with 28% higher mean quality score with larger diversity and without the mode collapse issue. Unlike typical generative models that usually generate new designs by interpolating within the boundary of training data, we show that PaDGAN expands the design space boundary outside the training data towards high-quality regions. The proposed method is broadly applicable to many tasks including design space exploration, design optimization, and creative solution recommendation.

scholarly journals RF-GANs: A Method to Synthesize Retinal Fundus Images Based on Generative Adversarial Network

2021 ◽  
Vol 2021 ◽  
pp. 1-17
Author(s):  
Yu Chen ◽  
Jun Long ◽  
Jifeng Guo
Keyword(s):  
Data Augmentation ◽  
Semantic Segmentation ◽  
Training Data ◽  
Generative Adversarial Networks ◽  
Fundus Images ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Retinal Fundus Images ◽  
Retinal Fundus ◽  
Segmentation Models

Diabetic retinopathy (DR) is a diabetic complication affecting the eyes, which is the main cause of blindness in young and middle-aged people. In order to speed up the diagnosis of DR, a mass of deep learning methods have been used for the detection of this disease, but they failed to attain excellent results due to unbalanced training data, i.e., the lack of DR fundus images. To address the problem of data imbalance, this paper proposes a method dubbed retinal fundus images generative adversarial networks (RF-GANs), which is based on generative adversarial network, to synthesize retinal fundus images. RF-GANs is composed of two generation models, RF-GAN1 and RF-GAN2. Firstly, RF-GAN1 is employed to translate retinal fundus images from source domain (the domain of semantic segmentation datasets) to target domain (the domain of EyePACS dataset connected to Kaggle (EyePACS)). Then, we train the semantic segmentation models with the translated images, and employ the trained models to extract the structural and lesion masks (hereafter, we refer to it as Masks) of EyePACS. Finally, we employ RF-GAN2 to synthesize retinal fundus images using the Masks and DR grading labels. This paper verifies the effectiveness of the method: RF-GAN1 can narrow down the domain gap between different datasets to improve the performance of the segmentation models. RF-GAN2 can synthesize realistic retinal fundus images. Adopting the synthesized images for data augmentation, the accuracy and quadratic weighted kappa of the state-of-the-art DR grading model on the testing set of EyePACS increase by 1.53% and 1.70%, respectively.

scholarly journals Building Extraction from High-Resolution Aerial Imagery Using a Generative Adversarial Network with Spatial and Channel Attention Mechanisms

2019 ◽  
Vol 11 (8) ◽  
pp. 917 ◽  
Author(s):  
Xuran Pan ◽  
Fan Yang ◽  
Lianru Gao ◽  
Zhengchao Chen ◽  
Bing Zhang ◽  
...  
Keyword(s):  
Remote Sensing ◽  
High Resolution ◽  
Ground Truth ◽  
Semantic Segmentation ◽  
Aerial Image ◽  
Remote Sensing Images ◽  
Generative Adversarial Network ◽  
Practical Applications ◽  
Image Labeling ◽  
Adversarial Network

Segmentation of high-resolution remote sensing images is an important challenge with wide practical applications. The increasing spatial resolution provides fine details for image segmentation but also incurs segmentation ambiguities. In this paper, we propose a generative adversarial network with spatial and channel attention mechanisms (GAN-SCA) for the robust segmentation of buildings in remote sensing images. The segmentation network (generator) of the proposed framework is composed of the well-known semantic segmentation architecture (U-Net) and the spatial and channel attention mechanisms (SCA). The adoption of SCA enables the segmentation network to selectively enhance more useful features in specific positions and channels and enables improved results closer to the ground truth. The discriminator is an adversarial network with channel attention mechanisms that can properly discriminate the outputs of the generator and the ground truth maps. The segmentation network and adversarial network are trained in an alternating fashion on the Inria aerial image labeling dataset and Massachusetts buildings dataset. Experimental results show that the proposed GAN-SCA achieves a higher score (the overall accuracy and intersection over the union of Inria aerial image labeling dataset are 96.61% and 77.75%, respectively, and the F1-measure of the Massachusetts buildings dataset is 96.36%) and outperforms several state-of-the-art approaches.

scholarly journals Dialectical GAN for SAR Image Translation: From Sentinel-1 to TerraSAR-X

2018 ◽  
Vol 10 (10) ◽  
pp. 1597 ◽  
Author(s):  
Dongyang Ao ◽  
Corneliu Octavian Dumitru ◽  
Gottfried Schwarz ◽  
Mihai Datcu
Keyword(s):  
Sar Image ◽  
Sar Images ◽  
High Quality ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Image Translation ◽  
Gram Matrices ◽  
Processing Resources ◽  
Ground Coverage

With more and more SAR applications, the demand for enhanced high-quality SAR images has increased considerably. However, high-quality SAR images entail high costs, due to the limitations of current SAR devices and their image processing resources. To improve the quality of SAR images and to reduce the costs of their generation, we propose a Dialectical Generative Adversarial Network (Dialectical GAN) to generate high-quality SAR images. This method is based on the analysis of hierarchical SAR information and the “dialectical” structure of GAN frameworks. As a demonstration, a typical example will be shown, where a low-resolution SAR image (e.g., a Sentinel-1 image) with large ground coverage is translated into a high-resolution SAR image (e.g., a TerraSAR-X image). A new algorithm is proposed based on a network framework by combining conditional WGAN-GP (Wasserstein Generative Adversarial Network—Gradient Penalty) loss functions and Spatial Gram matrices under the rule of dialectics. Experimental results show that the SAR image translation works very well when we compare the results of our proposed method with the selected traditional methods.

scholarly journals Scene Understanding Based on High-Order Potentials and Generative Adversarial Networks

2018 ◽  
Vol 2018 ◽  
pp. 1-8
Author(s):  
Xiaoli Zhao ◽  
Guozhong Wang ◽  
Jiaqi Zhang ◽  
Xiang Zhang
Keyword(s):  
Scene Understanding ◽  
Ground Truth ◽  
Semantic Segmentation ◽  
High Order ◽  
Generative Adversarial Networks ◽  
Class Label ◽  
Adversarial Network ◽  
Adversarial Networks ◽  
Segmented Image ◽  
Segmentation Framework

Scene understanding is to predict a class label at each pixel of an image. In this study, we propose a semantic segmentation framework based on classic generative adversarial nets (GAN) to train a fully convolutional semantic segmentation model along with an adversarial network. To improve the consistency of the segmented image, the high-order potentials, instead of unary or pairwise potentials, are adopted. We realize the high-order potentials by substituting adversarial network for CRF model, which can continuously improve the consistency and details of the segmented semantic image until it cannot discriminate the segmented result from the ground truth. A number of experiments are conducted on PASCAL VOC 2012 and Cityscapes datasets, and the quantitative and qualitative assessments have shown the effectiveness of our proposed approach.

scholarly journals Dialectical GAN for SAR Image Translation: From Sentinel-1 to TerraSAR-X

2018 ◽  
Author(s):  
Dongyang Ao ◽  
Corneliu Octavian Dumitru ◽  
Gottfried Schwarz ◽  
Mihai Datcu
Keyword(s):  
Electromagnetic Spectrum ◽  
Sar Image ◽  
Sar Images ◽  
High Quality ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Optical Images ◽  
Image Translation ◽  
Gram Matrices ◽  
Processing Resources

Contrary to optical images, Synthetic Aperture Radar (SAR) images are in different electromagnetic spectrum where the human visual system is not accustomed to. Thus, with more and more SAR applications, the demand for enhanced high-quality SAR images has increased considerably. However, high-quality SAR images entail high costs due to the limitations of current SAR devices and their image processing resources. To improve the quality of SAR images and to reduce the costs of their generation, we propose a Dialectical Generative Adversarial Network (Dialectical GAN) to generate high-quality SAR images. This method is based on the analysis of hierarchical SAR information and the “dialectical” structure of GAN frameworks.  As a demonstration, a typical example will be shown where a low-resolution SAR image (e.g., a Sentinel-1 image) with large ground coverage is translated into a high-resolution SAR image (e.g., a TerraSAR-X image). Three traditional algorithms are compared, and a new algorithm is proposed based on a network framework by combining conditional WGAN-GP (Wasserstein Generative Adversarial Network - Gradient Penalty) loss functions and Spatial Gram matrices under the rule of dialectics. Experimental results show that the SAR image translation works very well when we compare the results of our proposed method with the selected traditional methods.

Sign in / Sign up

Export Citation Format

Share Document